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Welcome  to  the  NDIA  Systems  Engineering  Conference 

On  behalf  of  the  National  Defense  Industrial  Association’s  Systems  Engineering  Division,  I  would  like  to 
extend  a  very  warm  welcome  to  the  20^^  Annual  Systems  Engineering  Conference.  Yes,  the  20^^  Annual 
-  who  knew  when  we  started  this  conference  2  decades  ago  that  we  would  continue  to  have  important 
systems  engineering  issues  to  address?  Well,  perhaps  most  of  you  -  because  after  all,  technology  keeps 
moving,  our  military  capability  continues  to  increase,  the  complexity  of  our  systems  continues  to  grow,  and 
the  threats  we  have  to  address  continue  to  grow  at  an  alarming  rate. 

For  example,  20  years  ago  the  term  “Cybersecurity”  wasn’t  addressed  in  DoD  circles.  Interoperability 
wasn’t  considered.  Systems-of-systems  weren’t  mentioned.  And  today,  these  are  some  of  our  hottest 
issues  that  the  entire  defense-industrial  complex  seeks  to  successfully  address,  not  to  mention  affordability, 
sustainability  and  a  host  of  other  issues  that  continue  to  need  attention. 

This  conference  is  the  primary  one  in  the  US  that  brings  together  the  engineering  arms  of  the  Cffice  of  the 
Secretary  of  Defense,  the  Services,  many  of  the  Federal  Agencies,  and  the  defense  industrial  complex  to 
address  and  seek  solutions  to  the  issues  we  all  face.  Executives,  managers  and  engineers  from  all  of  the 
major  US  defense  contractors,  as  well  as  the  principal  engineering  executives,  managers  and  engineers 
from  the  Department  of  Defense  and  the  Services  and  Federal  Agencies  are  here,  and  dialog  among  us 
is  critical  to  achieving  a  mutual  understanding  of  the  issues  we  collectively  face  and  desperately  need  to 
solve.  This  conference  provides  an  outstanding  opportunity  to  have  that  dialog  and  exchange  ideas,  so 
please  take  maximum  advantage  of  this  opportunity. 

And  if  there  is  anything  that  the  conference  committee,  whose  names  are  listed  in  the  program,  or  I,  or  the 
outstanding  NDIA  staff  can  do  to  assist  you,  please  let  us  know. 


Bob  Rassa 

Manager,  Engineering  Programs 
Raytheon  Space  &  Airborne  Systems 
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Dear  Attendees,  Speakers  and  Sponsors, 

I  would  like  to  add  my  warm  welcome  to  those  attending  the  annual 
Systems  Engineering  Division  conference.  This  year’s  conference  marks 
the  20th  anniversary  of  this  prestigious  event.  I  congratulate  the  NDIA 
Systems  Engineering  Division  for  their  sustained,  superior  performance  in 
producing  a  highly  consequential  event  and  applaud  the  many  ways  the 
division  supports  the  Defense  Department  and  defense  community. 

This  conference  is  the  premier  event  addressing  the  application  of  systems 
engineering  principles  to  defense  acquisition.  As  such,  it  is  the  main  forum 
to  exchange  information  and  ideas  among  the  Defense  Department,  the 
services,  defense  agencies,  industry  and  academia. 

I  wish  the  best  of  experiences  here  at  the  conference,  and  look  forward  to  many  more  years  of  division 
engagement  with  the  community  to  promote  and  refine  the  systems  engineering  practice. 

Sincerely 


Herbert  J.  Carlisle 
General,  USAF  (Ret) 
President  and  CEO 
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20™  ANNUAL  SYSTEMS  ENGINEERING 
CONFERENCE 

OCTOBER  23-26,  201 7  |  SPRINGFIELD,  VA 


INTRODUCTION 

Considered  the  major  annual  systems  engineering  event  foousing  on  the  performanoe  of  DoD  programs  and 
systems,  the  National  Defense  Industrial  Association’s  Annual  Systems  Engineering  Conference  offers  content 
tailored  to  all  levels  of  systems  engineering  (SE)  professionals: 

•  Keynote  Presentation 

•  Systems  Engineering  Executive  Panels 

•  DoD  Executive  Panel:  Service  Systems  Engineering  Leads  discuss  SE  issues 

•  DoD  Executive  Panel:  Interagency  Systems  Engineering  Activity 

•  Industry  Executive  Panel:  Industry  Leaders  discuss  Systems  Engineering  issues 

•  DoD  Executive  Panel:  Service  and  Agency  Program  Managers  discuss  systems  engineering  issues 

•  Technical  Breakout  Sessions  (2+  days) 

Demonstrating  broad  systems  engineering  community  support,  the  conference  is  once  again  this  year  enjoying 
technical  co-sponsorship  by  IEEE  AES,  IEEE  Systems  Council  and  the  International  Council  on  Systems 
Engineering. 

Further  attesting  to  its  value  and  relevance  to  Systems  Engineering  professionals  within  the  defense  industry, 
the  conference  continues  to  receive  the  support  of  the  Office  of  the  Deputy  Assistant  Secretary  of  Defense  for 
Systems  Engineering. 

Major  themes  running  through  the  three  plus  day  agenda  will  include  net-centric  operations,  data/information 
interoperability,  system-of-systems  engineering,  cyber  security  and  all  aspects  of  system  sustainment. 


CONFERENCE  OBJECTIVE 

This  conference  seeks  to  create  an  interactive  forum  for  Program  Managers,  Systems  Engineers,  Chief  Scientists, 
Engineers,  and  Managers  from  the  Requirements,  Design,  Verification,  Support,  Logistics  and  Test  communities 
from  both  government  and  industry.  The  conference  and  the  professional  exchanges  it  will  prompt  will  create 
opportunities  to  shape  future  policy  and  procedures. 


4 


SYSTEMS  ENGINEERING  CONFERENCE 


BACKGROUND 

The  Department  of  Defense  continues  to  seek  ways  to  improve 
the  acquisition  of  military  equipment  and  capability  to  assist  the 
warfighter  in  protecting  the  U.S.  and  its  Allies  around  the  world  in  a 
complex  environment  of  ever-changing  threats  and  conditions. 

The  Weapon  Systems  Acquisition  Reform  Act  (WSARA)  of 
2009  defines  Systems  Engineering  as  a  key  enabler  to  effect 
Improvements  in  defense  acquisition  and  program  execution  that 
will  produce  more  effective  and  affordable  military  systems.  Previous 
DoD  Better  Buying  Power  initiatives,  with  their  focus  on  achieving 
dominant  capabilities  through  technical  excellence  and  Innovation, 
continued  to  emphasize  the  importance  of  engineering  to  the 
Department.  The  new  administration  seeks  to  increase  military 
spending  which  will  put  additional  onus  on  the  defense  Industrial 
complex  to  achieve  acquisition  excellence,  and  systems  engineering 
performance  on  the  part  of  government  and  industry  as  partners  is  a 
key  ingredient  to  success. 

Systems  Engineering  is  the  “umbrella”  engineering  function  that 
drives  successful  program  execution  and  ensures  an  appropriate 
balance  between  requirements,  performance,  cost,  schedule,  and 
overall  effectiveness  and  affordability.  Systems  Engineering  principles 
embody  strong  technical  and  risk/opportunity  management 
aspects  for  the  acquiring  Program  Office  as  well  as  the  prime 
and  subcontractors.  Strong  emphasis  on  systems  engineering 
throughout  a  program,  especially  in  early  development  planning,  is  a 
key  enabler  of  successfully  fielding  complex  defense  systems. 

NDIA’s  Annual  Systems  Engineering  Conference  explores  the  various 
roles  of  systems  engineering  from  all  aspects  and  perspectives— 
pragmatic,  practical  and  academic— and  brings  key  practitioners 
together  to  work  on  effective  solutions  to  achieve  a  successful  and 
affordable  warfighting  force. 


CONFERENCE  CHAIR 

Mr.  Robert  Rassa 
Director,  Engineering 
Programs 

Raytheon  Company 

DIVISION  CHAIR 

Mr.  Frank  Serna 
Principal  Director,  Strategic 
Initiatives 
Draper  Laboratory 

DIVISION  VICE-CHAIR 

Mr.  Joseph  Elm 
Director  of  Engineering 
L-3  Communications 

NDIA  PLANNING  TEAM 

Ms.  Tammy  Kicker,  CMP 
Director,  Meetings  &  Events 

Ms.  Tina  Fletcher 
Meeting  Planner,  Meetings 
&  Events 
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SCHEDULE  AT  A  GLANCE 


MONDAY, 

OCTOBER  23 

8:00  am  - 

12:00  pm 

12:00  pm 

-  5:30  pm 

1 :00  pm  - 

3:00  pm 

3:00  pm  - 

3:30  pm 

3:30  pm  - 

5:30  pm 

TUESDAY, 

OCTOBER  24 

7:00  am  - 

5:00  pm 

7:00  am  - 

8:15  am 

8:15  am  - 

8:30  am 

8:30  am  - 

9:30  am 

9:30  am  - 

10:00  am 

10:00  am 

-  1 1 : 1 5  am 

1 1 : 1 5  am 

-  12:30  pm 

12:30  pm 

-1:30  pm 

1 :30  pm  - 

2:45  pm 

2:45  pm  - 

3:00  pm 

3:00  pm  - 

3:30  pm 

3:30  pm  - 

5:00  pm 

5:00  pm  - 

6:30  pm 

Display  Move  In 

Registration 

Tutorials 

Networking  Break 
Tutorials  continue 


Registration 
Networking  Breakfast 

Opening  Remarks:  Bob  Rassa,  Raytheon;  Frank  Serna,  Draper  Labs 

Plenary  Session  Keynote:  Vice  Admiral  Paul  Grosklags,  USN, 
Commander,  Naval  Air  Systems  Command 

Networking  Break 

Executive  Panel:  DoD  Systems  Engineering 
Executive  Panel:  Interagency  Systems  Engineering 
Networking  Luncheon 

Plenary  Session  Continues:  Industry  Executive  Panel 

Presentation  of  Lt  Gen  Thomas  R.  Ferguson  Systems  Engineering 
Excellence  Awards 

Networking  Break 

Executive  Panel:  Program  Managers 
Networking  Reception 
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WEDNESDAY  OCTOBER  25 


7:00  am  -  5:15  pm 

Registration 

7:00  am  -  8:00  am 

Networking  Breakfast 

8:00  am  -  9:40  am 

Conourrent  Breakout  Foous  Sessions  A 

9:40  am  -  10:15  am 

Networking  Break 

10:15  am  -  1 1 :55  am 

Concurrent  Breakout  Focus  Sessions  B 

1 1 :55  am  -  1 :00  pm 

Networking  Luncheon 

1 :00pm  -  2:40  pm 

Concurrent  Breakout  Focus  Sessions  C 

2:40  pm-  3:15  pm 

3:15  pm  -  5:20  pm 

Networking  Break 

Concurrent  Breakout  Focus  Sessions  D 

THURSDAY  OCTOBER  26 

7:00  am  -  5:15  pm 

Registration 

7:00  am  -  8:00  am 

Networking  Breakfast 

8:00  am  -  9:40  am 

Concurrent  Breakout  Focus  Sessions  A 

9:40  am  -  10:15  am 

Networking  Break 

10:15  am  -  1 1 :55  am 

Concurrent  Breakout  Focus  Sessions  B 

1 1 :55  am  -  1 :00  pm 

Networking  Luncheon 

1 :00  pm  -  2:40  pm 

Concurrent  Breakout  Focus  Sessions  C 

2:40  pm-  3:15  pm 

Networking  Break 

3:15  pm  -  5:20  pm 

Concurrent  Breakout  Focus  Sessions  D 
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TRACK  OBJECTIVES 

AGILE  IN  SYSTEMS  ENGINEERING 

Track  Chairs:  John  Norton,  Raytheon  Company 
Linda  Maness,  Northrop  Grumman  Corporation 
Eileen  Wrubel,  Software  Engineering  institute 

Agile  usage  is  becoming  more  prevalent  within  the  government 
space.  Lessons  learned  and  ideas  for  implementation  can  be  shared 
with  those  who  are  experienced  in  using  Agile  concepts.  This  track 
brings  together  practitioners  with  experience  applying  agile  methods 
in  a  variety  of  disciplines  and  domains,  with  the  goal  of  collaboration 
to  expand  their  effective  use  in  systems  engineering  and  on  defense 
programs 

ARCHITECTURE 

Track  Chairs:  Bob  Scheuer,  The  Boeing 
Ed  Moshinsky,  Lockheed  Martin  Corporation 

Architecture  is  a  key  element  in  systems  engineering.  This  track 
addresses  architecture  frameworks,  strategies,  and  applications  to 
improve  system  design,  test,  operations,  and  support. 

COMPUTATIONAL  RESEARCH  &  ENGINEERING 
ACQUISITION  TOOLS  AND  ENVIRONMENTS  (CREATE) 

Track  Chair:  Douglass  Post,  DoD  High  Performance  Computing 
Modernization  Program  (HPCMP) 

The  DoD  HPCMP  CREATE  Program  is  a  Tri-Service  Program 
launched  in  2006  by  OSD  and  the  HPCMP  to  develop  and  deploy 
eleven  physics-based  high  performance  computing  software 
applications  specifically  to  enable  the  DoD  acquisition  engineering 
community  to  design  and  analyze  military  ships,  aircraft,  ground 
vehicles,  and  radio  frequency  antennas.  These  tools  enable 
engineers  to  generate  an  arbitrarily  large  number  of  design  options 
(virtual  prototypes  expressed  as  digital  product  models)  for  design- 
space  exploration,  rapidly  assess  the  feasibility  and  performance 
characteristics  of  each  design  option,  and  accurately  predict  the 
performance  of  each  weapon  platform  with  high-fidelity  tools. 
With  these  tools,  DoD  engineers  can  identify  design  defects  and 
performance  shortfalls  and  fix  them  before  metal  has  been  cut, 
thus  reducing  costly  rework  and  improving  system  performance. 
This  reduces  the  cost,  schedule,  and  risk  of  acquisition  programs. 
The  tools  and  computer  time  are  available  to  DoD  engineers 
(government  and  industry).  The  tools  are  being  used  by  more  than 
180  DoD  engineering  organizations  (government  40%,  industry 
50%,  and  other  10%-including  academia)  with  over  1,400  users. 

DEVELOPMENTAL  TEST  &  EVALUATION  (DT&E) 

Track  Chairs:  Joe  Manas,  Raytheon  Company 

Developmental  Test  and  Evaluation  is  a  key  aspect  of  successful 
systems  engineering.  This  track  addresses  the  entire  continuum  of 
test  and  evaluation  from  early  planning  to  operational  testing. 

DIGITAL  ENGINEERING/MODEL-BASED  SYSTEMS 
ENGINEERING 

Track  Chair:  Philomena  Zimmerman,  DASD/SE 

Digital  Engineering  is  an  emerging  set  of  practices  for  Systems 
Engineering  and  other  engineering  disciplines  which  has,  at  its 
core,  the  use  of  models  (data,  algorithms  and/or  processes)  as  a 
technical  means  of  communication.  When  used  properly,  models 
can  provide  a  cohesion  across  engineering  activities,  and  cohesion 


with  acquisition  activities.  When  coupled  with  computational 
capabilities,  resultant  data  from  simulations  can  be  used  in 
decision-making  at  all  echelons,  and  an  increased  level  of  insight 
and  risk  reduction  in  the  end  item  can  be  achieved. 

ENGINEERED  RESILIENT  SYSTEMS  (ERS) 

Track  Chairs:  Lois  Hollan,  Potomac  institute 

Engineered  Resilient  Systems  (ERS)  is  a  Department  of  Defense 
priority  initiative  that  seeks  to  transform  engineering  environments 
so  that  warfighting  systems  are  more  resilient  and  affordable  across 
the  acquisition  lifecycle.  The  track  will  present  new  results  across 
the  ERS  initiative  including  anchor  technologies  and  computational 
representation. 

EDUCATION  &  TRAINING 

Track  Chair:  Don  Gelosh,  Worcester  Poiytechnic  Institute 

The  Education  and  Training  track  for  2017  is  an  excellent  collection 
of  thirteen  presentations  from  government,  industry,  and  academia. 
The  presentations  describe  a  wide  range  of  systems  engineering 
workforce  development  activities  from  competency  frameworks, 
cybersecurity  skills,  MBE  and  MBSE  best  practices.  System  of 
Systems  guide  and  capstone  marketplace  to  development  of 
technical  leaders. 

ENTERPRISE  HEALTH  MANAGEMENT/PROGNOSTICS/ 
DIAGNOSTICS/RELIABILTY 

Track  Chairs:  Chris  Resig,  The  Boeing  Company 

The  health  of  the  system  as  a  whole  -  the  enterprise  -  is  a  critical  function 
of  systems  engineering.  This  session  will  touch  on  some  issues  relating 
to  the  system  health,  including  prognostics,  diagnostics  and  reliability. 

ENVIRONMENT,  SAFETY,  AND  OCCUPATIONAL  HEALTH 
(ESOH) 

Track  Chairs:  Sherman  Forbes,  USAE 

Dave  Schulte,  SAiC 

Lucy  Rodriguez,  Booz  Alien  Hamilton 

The  ESOH  track  provides  a  cross  section  of  topics  that  reflect  the 
many  different  Systems  Engineering  design  considerations  included 
under  the  DoDI  5000.02  acronym  ESOH,  as  defined  in  MIL-STD- 
882E,  the  DoD  Standard  Practice  for  System  Safety.  This  year,  Mr. 
James  Thompson,  Director,  Major  Program  Support  (MPS),  within 
the  Office  of  the  Deputy  Assistant  Secretary  of  Defense  for  Systems 
Engineering  will  be  the  ESOH  track’s  keynote  speaker.  Mr.  Thompson 
will  share  his  perspectives  on  Risk,  Issue,  and  Opportunity  (RIO) 
Management  and  Independent  Technical  Risk  Assessments  (ITRAs). 
Mr.  David  Asiello,  the  Acquisition,  Sustainability  &  Technology 
Programs  lead  in  the  Office  of  the  Assistant  Secretary  of  Defense 
for  Energy,  Installations,  and  Environment  will  follow  Mr.  Thompson’s 
presentation  with  a  presentation  focusing  on  how  ESOH  Risk 
Management  is  an  integral  part  of  the  RIO  Management  Process 
and  offering  suggestions  for  improving  the  rigor,  accountability, 
and  visibility  of  ESOH  risk  management.  There  will  be  an  extended 
question  and  answer  period  following  Mr.  Thompson’s  and  Mr. 
Asiello’s  presentations  to  allow  the  audience  to  further  explore  the 
Acquisition  and  Sustainment  Risk  Management.  The  remainder  of 
the  ESOH  track  presentations  will  address  specific  acquisition  ESOH 
issues,  to  include  using  Digital  Engineering  to  manage  ESOH  risks 
and  requirements,  how  to  manage  ESOH  in  Rapid  Acquisitions, 
software  system  safety,  hazardous  materials  regulations  and 
management  impacts  on  programs,  environmental  liabilities, 
environmental  sustainability,  and  lessons  learned  about  program 
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office  successes  and  failures  in  implementing  the  DoDI  5000.02 
acquisition  ESOH  policy. 

HUMAN  SYSTEMS  INTEGRATION  (HSI) 

Track  Chair:  Matthew  Risser,  Pacific  Science 
Patrick  Fly,  The  Boeing  Company 

The  HSI  sessions  include  technical  papers  aligned  with  DoD  HSI  policy, 
standards  and  guidance.  The  goal  is  to  address  HSI  implications  in 
the  design  of  complex  systems  in  support  of  systems  engineering 
and  include  HSI  methods,  metrics,  and  best  practices,  process 
improvements,  applications  and  approaches  to  program  integration. 

INTEROPERABILITY/NET  -  CENTRIC  OPERATIONS 

Track  Chairs:  Jack  Zavin,  OUSD/ATL 
John  Daly,  Booz-Aiien-Hamiiton 

Interoperability  is  ability  to  operate  in  synergy  in  the  execution  of 
assigned  tasks  both  within  the  DoD  and  its  external  mission  partners. 
Net  Centric  Operations  supports  interoperability  by  providing  the 
POPIM  solution  sets  that  allows  the  DoD  and  its  mission  partners 
to  share  information/data/knowledge  when  needed,  where  needed, 
and  in  a  form  they  can  understand  and  act  on  with  confidence, 
while  protecting  it  from  those  who  should  not  have  it.  Net  Centric 
Operations/Interoperability  includes  technologies  such  as  Service 
Oriented  Architecture,  Data  Center,  Cloud  Computing,  information 
transport  [e.g.  internet,  web,  radios,  data  links],  as  well  as  both 
hardware  and  software  [aka  Information  and  Communicative 
Technology]  together  with  people,  operating  alone  or  in  organizations, 
as  part  of  the  System  of  Systems  Systems  Engineering. 

MISSION  ENGINEERING 

Track  Chair:  Judith  Dahmann,  MiTRE 

Mission  engineering  (ME)  is  the  deliberate  planning,  analyzing, 
organizing,  and  integrating  of  current  and  emerging  operational 
and  system  capabilities  to  achieve  desired  warfighting  mission 
effects.  This  track  focuses  on  current  directions  in  Defense  ME  and 
approaches  to  applying  SoS  and  SE  approach  to  ME. 

MODELING  AND  SIMULATION  (M&S) 

Track  Chairs:  David  Allsop,  The  Boeing  Company 
Chris  Schreiber,  Lockheed  Martin  Corpration 

The  M&S  Track  highlights  the  use  of  models  and  simulations  in 
the  systems  engineering  process.  Included  are  presentations  on 
integrated  environments,  tools  &  technologies,  and  M&S  applications 
in  several  SE  process  phases.  Topics  focused  specifically  on  Digital 
Engineering/Model-based  Systems  Engineering  are  contained  in  a 
separate  track  on  this  topic. 

PROGRAM  MANAGEMENT 

Track  Chairs:  Ken  Nidiffer,  Software  Engineering  institute 

Program  Managers  and  chief  Systems  Engineers  should  be  the 
“joined-at-the-hip”  leads  on  all  programs  that  wish  to  be  successful. 
This  session  will  address  some  of  the  issues  that  our  program 
managers  face  in  the  execution  of  programs. 


SOFTWARE  ENGINEERING 

Track  Chairs:  Ken  Nidiffer,  Software  Engineering  institute 

Software  is  often  overlooked  when  talking  systems  engineering  yet 
software  is  a  key  element  of  most  designs  today  and  must  always  be 
part  of  the  systems  engineer’s  portfolio  of  responsibility.  This  session 
will  highlight  a  few  significant  software  development  issues. 

SYSTEMS  ENGINEERING  EFFECTIVENESS 

Track  Chairs:  Tim  White,  Raytheon  Company 
Joe  Elm,  L3  Technoiogies 

Systems  Engineering  Effectiveness  is  obvious  to  some  and  quite 
esoteric  to  others.  The  goal  though,  improving  the  value  obtained 
for  each  SE  dollar  spent,  is  shared  by  each  who  joins  the  discussion. 
Please  attend  the  SE  Effectiveness  track  to  learn  how  your  peers  are 
implementing  practical  measures  to  better  quantify  the  benefits  of 
Systems  Engineering  and  its  value  to  Product  Users  and  Developers 
alike.  Early  and  effective  Systems  Engineering  has  been  shown  to 
return  excellent  value  to  all  project  stakeholders.  This  Track  will 
highlight  the  latest  DoD  policy  and  guidance,  define  new  approaches, 
and  provide  some  practical  experiences  to  assist  the  DoD  and 
defense  industry  SE  community  in  achieving  a  quantifiable  and 
persistent  improvement  in  program  outcomes  through  appropriate 
application  of  systems  engineering  principles  and  best  practices. 


SYSTEMS  OF  SYSTEMS  (SOS) 

Track  Chairs:  Judith  Dahmann,  MiTRE 
Rick  Poel,  The  Boeing  Company 
Jennie  Horn,  Raytheon  Company 

The  System  of  Systems  track  will  feature  papers  highlighting 
development  SoS  engineering  approaches,  particular  SoS  SE 
application  areas,  and  SoS  tools  and  modeling,  including  SoS  SE 
applied  to  defense  missions  in  mission  engineering.  See  directly 
related  track  in  Mission  Engineering,  above. 

SYSTEM  SECURITY  ENGINEERING  (SSE) 

Track  Chairs:  Holly  Dunlap,  Raytheon  Company 
Melinda  Reed,  DASD/SE 

System  Security  Engineering  has  become  one  of  the  most  important 
aspects  in  the  design  of  DoD  systems.  This  track  will  focus  on  system 
security  engineering  and  a  holistic  approach  to  program  protection. 
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SYSTEMS  ENGINEERING  CONFERENCE 


Monday,  October  23 


8:00am  -  12:00pm 
12:00pm  -  5:30pm 
1 :00  PM  -  5:30  pm 


Display  Move  In 
Registration  Open 
Tutorials 


3:00pm  -  3:30pm  Networking  Break 


3:30pm  -  4:00pm  4:00pm  -  4:30pm  4:30pm  -  5:00pm  5:00pm  -  5:30pm 

Track  4 

Gibson 

Tutorial: 
Modeling  and 
Simulation 
(M&S)  Cont’d 

1D4 

19696 

Half-Day  Tutorial:  Modeling  and  Simulation  in  the  Systems  Engineering  Process 
►  Dr.  Jim  Coolahan,  Coolahan  Consultants,  LLC 

Track  5 

Seller 

Tutorial: 
Applying  MIL- 
STD  Cont’d 

1D5 

19702 

Tutorial:  Applying  Focused  MIL-STD-882E  Software  Safety  Level  of  Rigor 
►  Mr.  Stuart  Whitford,  Booz  Allen  Hamilton 

Track  6 

Korman 

Tutorial: 
Communication 
and  Analysis 
Cont’d 

1D6 

19713 

Effective  Communication  and  Analysis  in  the  Age  of  MBSE 
►  Mr.  Ronald  Kratzke,  Vltech  Corporation 

5:30pm  Adjourn 
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SYSTEMS  ENGINEERING  CONFERENCE 


Tuesday,  October  24 


7:00am  -  5:00pm 
7:00am  -  8:15am 
8:15am  -  8:30am 


8:30am  -  9:30am 

9:30am  -  10:00am 
10:00am  -  11:15am 


Registration  Open 
Networking  Breakfast 
Opening  Remarks 

Mr.  Robert  Rassa,  Director,  Engineering  Progranns,  Raytheon  Company;  NDiA  Systems  Engineering  Conference 
Chair 

Mr.  Frank  Serna,  Principai  Director,  Strategic  initiatives.  Draper  Laboratory;  Chair,  NDiA  Systems  Engineering  Division 

Keynote  Presentation 

VADM  Paul  Grosklags,  NAVAiR,  Commander,  Navai  Air  Systems  Command 

Networking  Break 

DoD  Executive  Panel:  DoD  Systems  Engineering 

Moderator:  Mrs  Kristen  Baldwin,  Deputy  Assistant  Secretary  of  Defense,  Systems  Engineering  (Acting) 

Panelists: 

•  Col  Laird  Abbott,  USAE,  Chief,  Engineering  and  Eorce  Management  Division,  Deputy  Assistant  Secretary  for 
Science,  Technoiogy  and  Engineering,  SAE-AQR 

•  Mr.  William  Bray,  USN,  DASN  RDT&E  and  Chief  Systems  Engineer 

•  Mr.  Douglas  Wiltsie,  USA,  Executive  Director,  SoSE&l,  ASA  ALT  (invited) 


1 1  :15am  -  12:30pm 


12:30pm  -  1  :30pm 
1  :30pm  -  2:45pm 


2:45pm  -  3:00pm 
3:00pm  -  3:30pm 
3:30pm  -  5:00pm 


Executive  Panel:  Interagency  Systems  Engineering 

Moderator:  Ms.  Kristen  Baldwin,  Deputy  Assistant  Secretary  of  Defense,  Systems  Engineering  (Acting) 

Panelists: 

•  Mr.  Albert  “Benjie”  Spencer,  Nationai  Oceanic  and  Atmospheric  Administration 

•  Mr.  Jon  Holladay,  Technicai  Eeiiow  for  Systems  Engineering,  Nationai  Aeronautics  and  Space  Admnistration 

•  Mr.  Kent  Jones,  Assistant  Deputy  Administrator  for  Systems  Engineering  and  Integration,  Defense  Programs, 
DOE  Nationai  Nuciear  Security  Administration 

•  Mr.  Joseph  Post,  Deputy  Director,  NAS  Systems  Engineering  &  integration  Eederai  Aviation  Administration 

•  Mr.  James  Tuttle,  Deputy  Director,  CDS  and  Chief  Systems  Engineering,  Department  of  Homeiand  Security 

Networking  Luncheon 

Industry  Executive  Panel:  Model-Based  Systems  Engineering:  How  is  it  Helping? 

Mr.  Frank  Serna,  Principai  Director,  Strategic  Initiatives,  Draper  Laboratory;  Chair,  NDiA  Systems  Engineering  Division 

Panelists: 

•  Ms.  Christi  Gau  Pagnanelli,  Director,  BDS  Systems  Engineering  and  Engineering  Muiti-Skiiied  Leadership, 

Boeing  Defense,  Space  &  Security 

•  Mr.  Randall  Lum,  Corporate  Director,  Engineering,  Northrop  Grumman  Corporation 

•  Mr.  Tim  Walden,  Chief  Engineer  and  Eeiiow,  Lockheed  Martin  Corporate  Engineering  and  Production  Operations 

•  Mr.  Scott  Welles,  Vice  President,  Booz  Alien  Hamilton 

Presentation  of  Lt  Gen  Thomas  R.  Ferguson  Systems  Engineering  Excellence  Awards 
Networking  Break 

Executive  Panel:  Program  Managers 
Moderator:  Col.  David  Mclilece,  USAF 

Panelists: 

•  Col  Edward  Hospodar,  USAF,  GPS  User  Equipment  Senior  Materiel  Leader 

•  COL  Mike  Milner,  USA,  Armored  Multi-Purpose  Vehicle  (AMPV)  Program  Manager 

•  Col  Amanda  Myers,  USAF,  Deputy  Director,  Global  Reach  Programs,  Eormer  C-17  System  Program  Manager 

•  CAPT  Seiko  Okano,  USN,  PEC  Integrated  Wardare  Systems  (IWS)  2.0  Program  Manager 


5:00pm  -  6:30pm  Networking  Reception 
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SYSTEMS  ENGINEERING  CONFERENCE 


Wednesday,  October  25 

7:00am-5:15pm  Registration 


7:00am-8:00am  Networking  Breakfast 


8:00am  -  8:25am 

8:25am  -  8:50am 

8:50am  -  9:15am 

9:15am  -  9:40am 

Track  1 

Singleton 

Human  Systems 
Integration 

3A1 

19516 

Enhancing  Future  Soldier 
Systems  through  the  use 
of  the  Systems  Modeling 
Language  to  Incorporate 
Human  Aspects  into 
the  Soldier  as  a  System 
Definition 

►  Mr.  Sean  Pham,  U.S. 
ArmyARDEC 

19641 

HSI  Best  Practice  Standard 

►  Dr.  Patrick  Fly, 

The  Boeing  Company 

19739 

The  Human  Systems 
Integration  Partnership:: 
Delivering  the  HSI  Capability 
to  the  Air  Force  Systems 
Engineering  Process 

►  Mr.  Derek  Johnston, 

United  States  Air  Eorce 

19919 

Adaptive  Automation  for 

UAV  Pilot  Vehicle  Interfaces 

►  Mr.  Jeff  O’Hara,  Georgia 
Tech  Research  Institute 

Track  2 

DC 

LJJ 

—I 

Net  Centric 
Operations  & 
Interoperability 

3A2 

19752 

Kick  Off/Context  for  NCO/I 
Track 

►  Mr.  Jack  Zavin, 
DoD/OUSD(AT&L) 

19815 

ISO/IEC/IEEE8  15288 

System  Interoperability 
Considerations 

►  Mr.  John  Daly,  Booz  Alien 
Hamilton 

19759 

JITC  Executes  DoD  Mobility 
Field  Assessments 

►  Mr.  Khoa  Hoang,  Joint 
Interoperability  Test 
Command 

19764 

Interface  Management  for 
Interoperability- from  Theory 
to  Modeling 

►  Mr.  Matthew  Hause,  PTC 

Track  3 

Von  Sternberg 

Engineering  & 
Model-based 
Systems 
Engineering 

3A3 

19819 

DoD  Digital  Engineering 
Strategy 

►  Ms.  Philomena 

Zimmerman,  Department  of 
Defense 

19879 

Model  Centric  Engineering 
Enabling  a  New  Operational 
Paradigm  for  Acquisition 

►  Dr.  Mark  Blackburn, 
Stevens  Institute  of 
Technology 

19853 

Joint  NDIA  SSE  &  SwA 
Committee  and  Joint 
Federated  Assurance 

Center,  Government  SwA 

Gap  Analysis  Workshop 
Summary 

►  Ms.  Holly  Dunlap, 

Raytheon  Company 

19855 

MBSE  and  Systems 
Engineering  Transformation 

►  Mr.  Troy  Peterson, 

INCOSE 

Track  4 

Gibson 

Modeling  & 
Simulation 

3A4 

19691 

An  Autonomous  Sensor 
Tasking  System 

►  Ms.  Quintina  Jones, 
Raytheon  Missile  Systems 

19711 

Best  Practices  for  the 
Architecture,  Design,  and 
Modernization  of  Defense 
Models  and  Simulations 

►  Mr.  Michael  Heaphy, 
AT&L/DMSCO 

19725 

W8A  of  Models  and 
Simulations:  The  Power  of 
Independent  Cumulative 
Analyses 

►  Ms.  Natalie  Plotkin, 
Raytheon  Company 

19916 

Formalized  Execution  of 

Model  Integrated  Descriptive 
Architecture  Languages 

►  Mr.  Gregory  Haun, 
Analytical  Graphics,  Inc. 

Track  5 

Seller 

Agile 

3A5 

19877 

Research  Gone  “Agile”  A 

Case  Study  on  Using  an 
Enterprise  Transformation 
Process  to  Enable  Agile 
Methods  in  a  Research 
Program 

►  Dr.  Rosa  Heckle,  The 
MITRE  Corporation 

19726 

Issues  anOpportunities 
in  Accelerated  Software 
Development  for  Next 
Generation  DoD  Applications 

►  Dr.  Craig  Arndt, 

Defense  Acquisition 

University 

19755 

A  System  Dynamics 

Model  of  the  Scaled  Agile 
Framework  (SAFe)  to 

Quantify  the  Effects  of 
Management  Decisions  on 
Capability  Development  and 
Acquisition  Outcomes 

►  Mr.  Sean  Ricks,  The 

MITRE  Corporation 

19777 

“Elicitation  of  Robust  and 
Quality  Agile  User  Stories 
Using  QFD” 

►  Ms.  Sabrina  Ussery, 

The  George  Washington 
University 

Track  6 

Korman 

Software 

3A6 

19745 

Software  Complexity 

Modeling 

►  Mr.  Thuc  Tran, 

Capital  One 

19749 

Harnessing  the  Beast:  Using 
Model  Based  Systems 
Engineering  (MBSE)  to 
Manage  Complex  Research 
Software  Environments 

►  Ms.  Jennifer  Turgeon, 
Sandia  National 

Laboratories 

19758 

Software  Systems  Maturity 
Analysis 

►  Mr.  Christopher 

Dieckmann,  Idaho  National 
Laboratory 

19816 

Free  and  Open  Source 

Tools  to  Assess  Software 
Reliability  and  Security 

►  Mr.  Lance  Fiondella, 
University  of  Massachusetts 
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SYSTEMS  ENGINEERING  CONFERENCE 


Wednesday,  October  25  -  Continued 


9:40am-10:15am  Networking  Break 


10:15am  -  10:40am 

10:40am  -  11:05am 

11:05am  -  11:30am 

11:30am  -  11:55am 

Human  Systems 
Integration 

19784 

A  Wearable  Vision+lnertial 

19740 

Fisher  vs.  Taguchi 

19854 

NDIA  Welcome  and  Review 

19881 

DoD  Cyber  Resilient  Weapon 

Track  1 

Singleton 

Systems 

Security 

Engineering 

3B1 

Navigation  System  for 
Assessing  Volumetric 
Utilization  and  Task 

Geometry  Efficiency 

►  Mr.  Kevin  Duda, 

Draper  Laboratory 

Experimental  Design 

Methods  in  Human  Factors 

►  Ms.  Sarah  Ewing, 

Idaho  National  Laboratory 

of  Accomplishments 

►  Ms.  Holly  Dunlap, 

Raytheon  Company 

Systems 

►  Ms.  Melinda  Reed, 
Department  of  Defense 

C\J 

o 

DC 

LJJ 

—I 

Net  Centric 
Operations  & 
Interoperability 

19923 

Joint  and  Mission  Partner 
Interoperability 

19499 

Real  Life  Cloud  Acquisition 
and  Adoption  Across 

19849 

Mission  Integration 
Management,  NDAA  201 7 

19838 

Systems  of  Systems 
Engineering  Technical 

< 

A 

Mission 

Engineering 

3B2 

►  Mr.  Mike  Richards, 

Joint  Staff  J6 

Myenoies  anu  wiouu 

Providers 

►  Mr.  Mun-Wai  Hon,  Noblis 

oGGIIOr  1  oOO 

►  Mr.  Robert  Gold, 
Department  of  Defense 

ApprodGiieo  do  Appiieo  lO 

Mission  Engineering 

►  Dr.  Judith  Dahmann, 

MITRE 

CO 

o 

< 

A 

Von  Sternberg 

Digital 

Engineering  & 
Model-based 
Systems 
Engineering 

3B3 

19793 

Model-Centric  Decision 
Making:  Insights  from  an 
Expert  Interview  Study 

►  Dr.  Donna  Rhodes, 
Massachusetts  Institute  of 
Technology 

19890 

Using  MBSE  to 

Communicate  and  Gain 
Acceptance  of  your  Analysis 
►  Mr.  Frank  Salvatore, 

Engility 

19795 

New  Innovations  in  Digital 
Systems  Engineering 

►  Dr.  Edward  Kraft, 

University  of  Tennessee 
Space  Institute 

19920 

Key  MBSE  Enablers  with 
Examples 

►  Mr.  Nicholas  Driscoll,  III, 
Raytheon  Company 

Track  4 

Gibson 

CREATE 
Computational 
Research  & 
Engineering 
Acquisition 
Tools  and 
Environments 

3B4 

20010 

Digital  Engineering  (DE)  and 
Computational  Research 
and  Engineering  Acquisition 
Tools  and  Environments 
(CREATE) 

►  Ms.  Philomena 

Zimmerman, 

Department  of  Defense 

19721 

CREATE:  Accelerating 

Defense  Innovation  with 
Computational  Prototypes 
and  High  Performance 
Computers 

►  Dr.  Douglass  Post, 

DoD  HPCMP 

19730 

Physics-Based  Simulation 
in  Support  of  Acquisition 
program  and  Fleet 

Operations 

►  Mr.  Steven  Donaldson, 
Naval  Air  Systems 

Command 

19728 

Capstone:  A  Patform  for 
Geometry,  Meshing  and 
Attribution  Modeling  for 
Physics-based  Analysis  and 
Design 

►  Dr.  Saikat  Dey, 

US  NRL  Code  7131 

Agile 

19902 

Software  Development 
Challenges  in  AFMC  (Agile 
Software  Development  and 
Data  Rights) 

►  Mr.  Andrew  Jeselson,  Air 
Force  Materiel  Command 

19701 

Leveraging  Cybersecurity 
Tools  for  Software  Safety: 
Focusing  (Some)  Static 
Analysis  on  Safety-Critical 
Software 

►  Mr.  Stuart  Whitford, 

Booz  Allen  Hamilton 

20028 

Joint  Software  System 

Safety  Implementation  Guide 

►  Mr.  Bob  Smith, 

Booz  Allen  Hamilton 

Track  5 

Seller 

Environment 
Safety  & 
Occupational 
Health 

3B5 

Systems 

Engineering 

Effectiveness 

19850 

Engineering  Autonomy 

19882 

The  Drive  for  Innovation  in 

19814 

DoD  Systems  Engineering 

19835 

Helix:  Understanding 

CD 

z 

►  Mr.  Robert  Gold, 

Systems  Engineering 

Policy,  Guidance  and 

Systems  Engineering 

A 

<c 

DC 

O 

3B6 

Department  of  Defense 

►  Mr.  Scott  Lusero, 
Department  of  Defense 

Standardization 

►  Ms.  Aileen  Sedmak, 
Department  of  Defense 

Effectiveness  through 

Modeling 

►  Ms.  Nicole  Hutchison, 
Stevens  Institute  of 
Technology 

1 1  :55am  -  1  :00pm  Networking  Luncheon 
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SYSTEMS  ENGINEERING  CONFERENCE 


Wednesday,  October  25  -  Continued 


1:00pm  -  1:25pm 

1:25pm  -  1:50pm 

1:50pm  -  2:15pm 

2:15pm  -  2:40pm 

Track  1 

Singleton 

System  Security 
Engineering 

3C1 

19852 

NDIA  Cyber  Resilient  & 

Secure  Systems  Summit 
Summary 

►  Ms.  Holly  Dunlap, 

Raytheon  Company 

19839 

Unified  Architecture 
Framework  (UAF)  Profile 
for  Risk  Assessment 
Methodology 

►  Ms.  Tamara  Hambrick, 
Northrop  Grumman 
Corporation 

19913 

Considerations  to  Address 
Dependably  Secure 

System  Function  in  System 
Capability,  Requirements, 
and  Performance  Artifacts 

►  Mr.  Michael  McEvilley, 

The  MiTRE  Corporation 

19866 

AF  Cyber  Campaign  Plan  - 
Weapon  Systems  Focus 

►  Mr.  Daniel  Holtzman,  U.S. 
Air  Force 

C\J 

o 

< 

A 

Miller 

Mission 

Engineering 

System  of 
Systems 

3C2 

19706 

Model  Based  Systems  of 
Systems  Engineering 

►  Mr.  Francis  McCafferty, 
Vitech  Corporation 

19868 

Mission  Threads:  Linking 
Mission  Engineering  and 
Systems  Engineering 

►  Dr.  Greg  Butler, 

Engiiity  Corp 

19718 

Developing  Standards  for 
Systems  of  Systems  (SoS) 
Engineering 

►  Dr.  Judith  Dahmann, 

The  MiTRE  Corporation 

19804 

Scaling  Model-Based 

System  Engineering 

Practices  for  System  of 
Systems  Applications: 
Software  Tools 

►  Ms.  Janna  Kamenetsky, 
The  MiTRE  Corporation 

Track  3 

Von  Sternberg 

Digital 

Engineering  & 
Model-based 
Systems 
Engineering 

3C3 

19545 

Pulling  the  Digital  Thread 
with  Model  Based 

Engineering 

►  Mr.  Christopher  Finlay, 
Raytheon  Company 

19906 

Modeling  the  Digital  System 
Model  Data  Taxonomy 

►  Ms.  Philomena 
Zimmerman, 

Department  of  Defense 

19746 

Developing  and  Distributing 
a  CubeSat  Model-Based 
Systems  Engineering 
(MBSE)  Reference  Model  - 
Interim  Status  #2 

►  Dr.  David  Kaslow,  S.E.L.F 

19872 

Enabling  Design  of  Agile 
Security  with  MBSE 

►  Mr.  Barry  Papke, 

No  Magic 

Track  4 

Gibson 

CREATE: 
Computational 
Research  & 
Engineering 
Acquisition 
Tools  and 
Environments 
Engineering 

3C4 

19779 

High-Fidelity 

Electromagnetic  Modeling 
with  CREATE-RF  Tools 

►  Dr.  Daniel  Dault,  Air  Force 
Research  Lab 

19809 

Physics  Based  Modeling  & 
Simulation  For  Shock  and 
Vulnerability  Assessments 
-  Navy  Enhanced  Sierra 
Mechanics 

►  Mr.  Jonathan  Stergiou, 
Navai  Surface  Warfare 

Center,  Carderock  Division 

19823 

The  Role  of  CREATE-AV 
in  Realization  of  the  Digital 
Thread  “Authoritative  Truth 
Source” 

►  Dr.  Edward  Kraft, 

University  of  Tennessee 

Space  institute 

19753 

A  Networked  Frigate 

Concept  Design  Space 
Exploration  Using  the  Rapid 
Ship  Design  Environment 

►  Dr.  Douglas  Rigterink, 

Navei  Surface  Warfare 

Center,  Carderock  Division 

Track  5 

Seller 

Environment 
Safety  & 
Occupational 
Health 

3C5 

19912 

DASD  (SE)  Risk,  Issue, 
and  Opportunity  (RIO) 
Management  and 

Independent  Technical  Risk 
Assessments  (ITRAs) 

►  Mr.  James  Thompson, 
Department  of  Defense 

19697 

ESOH  Risk  Management 

►  Mr.  David  Asiello, 
OASD(Ei&E) 

19908 

DoD  Acquisition  ESOH  IPT  Q&A  Panel 

►  Mr.  David  Asiello, 

OASD(Ei&E) 

Track  6 

Korman 

Systems 

Engineering 

Effectiveness 

3C6 

19790 

Systems  Engineering 
Research  Needs  and 
Workforce  Development 

Study 

►  Dr.  Dinesh  Verma, 

Systems  Engineering 
Research  Center  (SERC) 

19744 

Technical  Performance  Risk 
Management  for  Large 

Scale  Programs 

►  Mr.  Brian  Davenport, 
Raytheon  Company 

19742 

The  Design  of  a  Cone 
Penetrometer  System 

►  Dr.  Doris  Turnage, 

U.  S.  Army  Engineer 
Research  &  Deveiopment 
Center 

19781 

Additive  Manufacturing  - 
Challenges  for  the  Systems 
Engineer  and  Program 
Manager 

►  Mr.  William  Decker, 

Defense  Acquisition 

University 
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SYSTEMS  ENGINEERING  CONFERENCE 


Wednesday,  October  25  -  Continued 


2:40pm  -  3:15pm  Networking  Break 


3:15pm  -  3:40pm 

3:40pm  -  4:05pm 

4:05pm  -  4:30pm 

Track  1 

Singleton 

System 

Security 

Engineering 

3D1 

19861 

Cyber  Resilient  and  Secure  Weapon 
Systems  Acquisition/Proposal 

Discussion  &  Summary 

►  Ms.  Holly  Dunlap,  Raytheon 

Company 

19771 

When  the  Right  Answer  is  Not  What 
NAVSEA  Normally  Does 

►  Mr.  Peter  Chu,  NAVSEA  05 

19870 

Can’t  We  Just  Get  Along:  Engineering 
Trade  Decisions  VS  RMF  at  the  System 
Level 

►  Mr.  Don  Davidson,  DoD  CIO 

Track  2 

DC 

LJJ 

—I 

System  of 
Systems 

3D2 

19802 

Scaling  Model-Based  System 

Engineering  Practices  for  System  of 
Systems  Applications:  Analytic  Methods 

►  Dr.  Aleksandra  Markina-Khusid, 

The  MITRE  Corporation 

19757 

Defense  System  of  Systems  Gap 

Analysis 

►  Mr.  Christopher  Dieckmann, 

Idaho  National  Laboratory 

19878 

Enterprise  Implications  of  Family  of 
Systems  (FoS)  Acquisition 

►  Dr.  Garrett  Thurston, 

Dassault  Systemes 

Track  3 

Von  Sternberg 

Digital 

Engineering  & 
Model -based 
Systems 
Engineering 

3D3 

19775 

Digital  System  Model  Ice 

►  Dr.  David  Hench, 

Eagle  Ray  R&D 

19871 

Enabling  Repeatable  SE  Cost 

Estimation  with  COSYSMO  and  MBSE 

►  Mr.  Barry  Papke,  No  Magic 

19888 

MBSE  to  Address  Logical  Text-Based 
Requirements  Issues 

►  Dr.  Saulius  Pavalkis, 

No  Magic 

Track  4 

Gibson 

CREATE: 
Computational 
Research  & 
Engineering 
Acquisition 
Tools  and 
Environments 
Engineering 

3D4 

19693 

Program  Management  in  CREATE 
for  the  Development  of  Large-scale 
Physics-based  Software  Development 
Projects  for  Engineering  Design  and 
Analysis 

►  Dr.  Richard  Kendall, 

DoD  HPCMP 

19704 

Computational  Research  and 
Engineering  Acquisition  Tools  and 
Environments  -  Ground  Vehicles 
(CREATE-GV) 

►  Dr.  Christopher  Goodin,  U.S.  Army 
ERDC 

19715 

Physics-based,  Multidisciplinary 

Analysis  of  Fixed-Wing  Aircraft  with 
HPCMP  CREATE(TM)-AV/Kestrel 

►  Dr.  David  McDaniel, 

DoD  HPCMP/CREATE 

Track  5 

Seller 

Environment 
Safety  & 
Occupational 
Health 

3D5 

19770 

Assessing  the  impacts  of  Amended  Toxic  Substances  Control  Act  to  the  DoD  Mission  and  the  Defense  Industrial  Base  Panel 

►  Ms.  Amy  Borman,  U.S.  Army 

►  COL  Joseph  Constantino  (SAE/IEE) 

►  Mr.  Shane  Esola,  DCMA 

►  Mr.  Jim  Rudroff,  (ODASN(E)) 

►  Dr.  Patricia  Underwood,  OASD(EI&E) 

Track  6 

Korman 

Systems 

Engineering 

Effectiveness 

3D6 

19738 

Improving  Effectiveness  with  respect 
to  Time-To-Market  and  the  Impacts  of 
Late-stage  Design  Changes  in  Rapid 
Development  Life  Cycles 

►  Mr.  Parth  Shah, 

George  Washington  University 

19716 

Integrity  System  Security  Engineering 
into  System  Engineering 

►  Mr.  Ken  Barker,  USAE 

19824 

Implementation  of  the  R&M  Engineering 
Body  of  Knowledge 

►  Mr.  Andrew  Monje, 

Department  of  Defense 
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SYSTEMS  ENGINEERING  CONFERENCE 


Wednesday,  October  25  -  Continued 


4:30pm  -  4:55pm 

4:55pm  -  5:20pm 

Track  1 

Singleton 

System  Security 
Engineering 

3D1 

19880 

Engaging  the  DoD  Enterprise  to 

Protect  U.S.  Military  Technical 
Advantage:  Joint  Acquisition  Protection 
and  Exploitation  Cell  Update 

►  Mr.  Brian  Hughes, 

Department  of  Defense 

19798 

Using  Real  Options  Analysis  to 
develop  Resiliency  in  System  Security 
Architectures 

►  Mr.  Chris  D’Ascenzo, 

Defense  Acquisition  University 

Track  2 

Miller 

System  of  Systems 

3D2 

19736 

“Defense  Acquisition  System”  System 
of  Systems  Engineering 

►  Mr.  Larry  Harding, 

Idaho  National  Laboratory 

Track  3 

Von  Sternberg 

Digital  Engineering 
&  Model- 
based  Systems 
Engoneering 

3D3 

19763 

The  Digital  Engineering  Journey 

►  Mr.  Mathew  Hause, 

PTC 

19833 

Digitalization  of  Systems  Engineering 
-Examples  and  Benefits  for  the 
Enterprise 

►  Mr.  Sanjay  Khurana, 

Dassault  Systemes 

Track  4 

Gibson 

CREATE: 
Computational 
Research  & 
Engineering 
Acquisition  Tools 
and  Environments 
Engineering 

3D4 

19776 

Weapons  System  Innovation  through 
Workflow-based  Computational 
Prototyping 

►  Mr.  Loren  Miller, 

DataMetric  Innovations,  LLC 

19786 

Rotorcraft  Acquisition:  Development  of 
Modeling  and  Simulation  Procedures 

►  Dr.  Marvin  Moulton, 

U.S.  Army 

Track  5 

Seller 

Environment 
Safety  & 
Occupational 
Health 

3D5 

19770 

Assessing  the  impacts  of  Amended  Toxic  Substances  Control  Act  to  the  DoD  Mission  and  the  Defense  Industrial  Base 

Panel 

►  Ms.  Amy  Borman,  U.S.  Army 

►  COL  Joseph  Constantino  (SAF/IEE) 

►  Mr.  Shane  Esola,  DCMA 

►  Mr.  Jim  Rudroff,  (ODASN(E)) 

►  Dr.  Patricia  Underwood,  OASD(EI&E) 

CD 

o 

< 

A 

Korman 

Systems 

Engineering 

Effectiveness 

3D6 

19762 

Decision-Driven  Product  Development 

►  Mr.  Matthew  Hause, 

PTC 

19830 

Are  We  Doing  Enough  in  Requirements 
Management? 

►  Dr.  Steven  Dam, 

SPEC  Innovations 

5:20pm  Adjourn 
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SYSTEMS  ENGINEERING  CONFERENCE 


Thursday,  October  26 

7:00am-5:15pm  Registration 


7:00am-8:00am  Networking  Breakfast 


8:00am  -  8:25am 

8:25am  -  8:50am 

8:50am  -  9:15am 

9:15am  -  9:40am 

Track  1 

Singleton 

System  Security 
Engineering 

4A1 

19796 

Cyber  Systems  Risk  -  an 
Opportunity  for  Model  Based 
Engineering  &  Design 

►  Dr.  Jerry  Couretas, 

Booz  Allen  Hamilton 

19785 

Cybersecurity  As  An  Integral 
Part  of  Systems  Engineering 

►  Mr.  William  Decker, 

Defense  Acquisition 

University 

19741 

Security  at  Design  Time: 
Addressing  Resilience  in 
Mission  Critical  Cyber- 
Physical  Systems 

►  Mr.  Thomas  McDermott, 

Jr.,  Georgia  Tech  Research 
Institute 

19911 

Achieving  DoD  Software 
Assurance  (SwA) 

►  Mr.  Thomas  Hurt, 
Department  of  Defense 

Track  2 

DC 

LJJ 

—I 

Developmental 
Test  & 
Evaluation 

4A2 

19792 

An  Approach  to  Verification 
of  Complex  Systems 

►  Dr.  Wilson  Felder, 

Stevens  Institute  of 
Technology 

19925 

Improving  Distributed  Testing 
with  TENA  and  JM ETC 

►  Mr.  Ryan  Norman, 
TENA/JMETC 

19774 

Identifying  Requirements 
and  Vulnerabilities  for 
Cybersecurity;  Or  How  1 
Learned  to  Stop  Worrying 
and  Love  the  Six-Phase 
Cybersecurity  T&E  Process 

►  Mr.  David  Brown, 

Electronic  Warfare 

Associates  (EWA) 

19831 

How  Can  We  Use  V&V 
Techniques  in  Early  Systems 
Engineering? 

►  Dr.  Steven  Dam, 

SPEC  Innovations 

Track  3 

Von  Sternberg 

Engineered 

Resilient 

Systems 

4A3 

20009 

Digital  Engineering  and  ERS 

►  Mr.  Robert  Gold, 
Department  of  Defense 

19845 

ERS:  Influencing  Acquisition 
Innovation 

►  Dr.  Owen  Esiinger, 

U.S.  Army  Engineer 

Research  and  Development 
Center 

19907 

Scaling  Data  Analytics  for 

ERS 

►  Mr.  David  Stuart, 

U.S.  Army  Engineer 

Research  and  Development 
Center 

Track  4 

Gibson 

Create: 

Computational 
Research  & 
Engineering 
Acquisition 
Tools  and 
Environments 
Engineering 

4A4 

19887 

Multi-Disciplinary  Integration 
of  ModSim  for  Navy 
Applications 

►  Dr.  Greg  Bunting, 

Sandia  National 

Laboratories 

19729 

Academic  Deployment  of  the 
HPCMP  CREATE  Genesis 
Software  Package 

►  Dr.  Robert  Meakin, 

U.S.  DoD  HPCMP 

19875 

Secure  Web-Based 

Access  for  Productive 
Supercomputing 

►  Ms.  Laura  Ulibarri, 

Air  Eorce  Research 
Laboratory 

19800 

CREATE-SH  IHDE:  Workflow 
Process  Improvements 
for  Hydrodynamics 
Characterization  of  Ship 
Designs 

►  Mr.  Wesley  Wilson,  Naval 
Surface  Warfare  Center, 
Carderock  Division 

Track  5 

Seller 

Environment, 
Safety  & 
Occupational 
Health 

4A5 

19773 

Model  Based  Systems 
Engineering  (MBSE) 
Considerations  for 
Environment  Safety  and 
Occupational  Health  (ESOH) 

►  Mr.  Leo  Kilfoy, 

MSC  Software 

19772 

A  Pragmatic  Approach  to 
System  Modeling  for  Hazard 
Identification  and  Risk 
Management 

►  Mr.  Michael  Vinarcik, 

Booz  Allen  Hamilton 

19708 

Unmanned  System  (UxS) 
Safety  Engineering  Precepts 
-  an  OSD  Guide  -  update  of 
the  2007  OSD  UxS  Safety 
Guide 

►  Mr.  Michael  Demmick, 
NOSSA 

19754 

Divergent  Oscillating 

Refueling  Probe  on  the 
HH-60G  Pavehawk 

►  Mr.  Joseph  Jones, 
SAE/AQRE 

Track  6 

Korman 

Architecture 

4A6 

19820 

MOSA  Considerations 
in  Systems  Engineering 
Through  the  Lifecycle 

►  Ms.  Philomena 

Zimmerman, 

Department  of  Defense 

19821 

Implementing  a  MOSA  to 
Achieve  Acquisition  Agility 
in  Defense  Acquisition 
Programs 

►  Ms.  Philomena 

Zimmerman, 

Department  of  Defense 

19837 

Challenges  to  Implementing 
MOSA  for  Major  DoD 
Acqusition  Programs 

►  Mr.  Edward  Mosh insky, 
Lockheed  Martin 

Corporation 

19778 

Investigating  Approaches  to 
Achieve  Modularity  Benefits 
in  the  Defense  Acquisition 
Ecosystems 

►  Dr.  Navindran 
Davendralingam, 

Purdue  University 
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SYSTEMS  ENGINEERING  CONFERENCE 


Thursday,  October  26-  Continued 


9:40am-10:15am  Networking  Break 


10:15am  -  10:40am 

10:40am  -  11:05am 

11:05am  -  11:30am 

11:30AM  -11:55AM 

Track  1 

Singleton 

System  Security 
Engineering 

4B1 

19853 

Joint  NDIA  SSE  &  SwA 
Committee  and  Joint 
Federated  Assurance 

Center,  Government  SwA 

Gap  Analysis  Workshop 
Summary 

►  Ms.  Holly  Dunlap, 

Raytheon  Company 

19698 

Program  Manager’s 
Guidebook  for  Integrating 
Software  Assurance  into 
Defense  Systems  During  the 
System  Acquisition  Lifecycle 

►  Dr.  Kenneth  Nidiffer, 
Software  Engineering 

Institute 

19735 

Reducing  Software 
Vulnerabilities  -  The  “Vital 

Few”  Process  and  Product 
Metrics 

►  Mr.  Girish  Seshagiri, 

Ishpi  Information 
Technologies,  Inc. 

19910 

DoD  Joint  Federated 

Assurance  Center  (JFAC)  201 7 
Update 

►  Mr.  Thomas  Hurt, 

Department  of  Defense 

Track  2 

Miller 

Education  & 
Training 

4B2 

19813 

Shaping  the  Department 
of  Defense  Engineering 
Workforce 

►  Ms.  Aileen  Sedmak, 
Department  of  Defense 

19794 

Review  of  Best  Practices 
for  Technical  Leadership 
Development 

►  Dr.  Wilson  Felder, 

Stevens  Institute  of 
Technology 

19805 

Development  of  a  Defense 
Mission  Engineering 
Competency  Model 

►  Dr.  Nicole  Hutchison, 
Stevens  Institute  of 
Technology 

19789 

The  Capstone  Marketplace: 
Growing  our  Technical 

Workforce  through  Systems 
Oriented  Senior  Design 

Projects 

►  Ms.  Megan  Clifford,  Systems 
Engineering  Research  Center 

Track  3 

Von  Sternberg 

Engineered 

Resilient 

Systems 

4B3 

19844 

Tradespace:  Informed 

Decision  making  for 
Acquisition 

►  Mr.  Timothy  Garton, 
Engineer  Research  and 
Development  Center 

19834 

Building  an  Agile  Framework 
for  the  Analysis  of 
Environmental  Impacts  on 
Military  Systems 

►  Dr.  Dharhas  Pothina, 
Engineer  Research  and 
Development  Center 

19859 

Introducing  Lifecycle 

Cost  to  Early  Conceptual 
Tradespace  Exploration 

►  Mr.  Erwin  Bay  lot. 

Engineer  Research 
and  Development 

Center 

19806 

Overcoming  the  Government  - 
Industry  Collaboration  Hurdle 

►  Dr.  Patrick  Martin, 

BAE  Systems 

Track  4 

Gibson 

Create: 

Computational 
Research  & 
Engineering 
Acquisition 
Tools  and 
Environments 
Engineering 

4B3 

19694 

Software  Engineering 
for  Physics-based  HPC 
Applications  for  Engineering 
Design  and  Analysis  in 
CREATE 

►  Dr.  Richard  Kendall,  DoD 
HPCMP 

19703 

Verification  and  Validation  in 
CREATE  Multi-Physics  HPC 
Software  Applications 

►  Dr.  Lawrence  Votta, 

Brincos  Inc. 

19709 

DoD  Risk  Management 
Deficiencies... And  How  to 

Fix  Them 

►  Mr.  Richard  Sugarman, 

U.S.  AirEorce 

19724 

Tools  for  Acquiring  Highly 
Maintainable  Software- Intensive 
Systems 

►  Dr.  Barry  Boehm,  USC 

Track  5 

Seller 

Environment, 
Safety  & 
Occupational 
Health 

4B5 

19767 

Rapid  Equipping  - 
Immediate  Need  to  Equip 
and  Protect  Soldiers 

►  Mr.  George  Evans, 
Prospective  Technology  Inc. 
(SAAL-PE/PTI  ctr) 

19769 

ESOH  Risk  Management 
and  Applying  MIL-STD- 
882E  Principles  to  Programs 
that  Deviate  from  Standard 
Acquisition  Models 

►  Mr.  Jefferson  Walker, 

Booz  Allen  Hamilton 

19732 

Hazardous  Materials 

Risk  Management  Using 
MIL-STD-882E 

►  Ms.  Lori  Hales, 

Booz  Allen  Hamilton 

19836 

Leveraging  the  International 
Aerospace  Environmental 

Group  (lAEG)  Defense 

Acquisition  Materials 

Declaration  Process 

►  Ms.  Karen  Gill, 

Booz  Allen  Hamilton 

Track  6 

Korman 

Architecture 

4B6 

19780 

Cybersecurity  and  a  Modular 
Open  Systems  Approach 

►  Mr.  William  Decker, 

Defense  Acquisition 

University 

19743 

If  System  Architectures  are 

So  Useful,  Why  Don’t  We 

Use  Them  More? 

►  Mr.  Robert  Scheurer,  NDIA 
SE  Architecture  Committee 

19873 

A  Reverse  Chronology  of 
Evolutionary  Architecture  and 
Agile  Development 

►  Mr.  Thomas  Mielke, 

CACI  International  Inc. 

19903 

Efficient  Use  of  Enterprise 
and  System  Architecting  in 
Combined  Environment 

►  Dr.  Howard  Gans, 

Harris  Corporation 
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SYSTEMS  ENGINEERING  CONFERENCE 


Thursday,  October  26  -  Continued 

1 1  :55am  - 1  :00pm  Networking  Luncheon 


1:00pm  -  1:25pm 

1:25pm  -  1:50pm 

1:50pm  -  2:15pm 

2:15pm  -  2:40pm 

Track  1 

Singleton 

System  Security 
Engineering 

4C1 

19862 

Long-Term  Strategy  for 

DoD  Trusted  and  Assured 
Microelectronics  Needs 

►  Dr.  Jeremy  Muldavin, 
Department  of  Defense 

19747 

SSE  Abstract:  Developing 
Trust  For  a  Secure 
Microelectronics  Supply 

Chain 

►  Dr.  Michael  Fritze, 

Potomac  Institute  for  Policy 
Studies 

19731 

SSE:  Trusted 

Microelectronics  Joint 

Working  Group 

►  Dr.  Brian  Cohen,  Institute 
for  Defense  Analyses 

19700 

Managing  Risk  with  Trusted 
ASICs:  Introducing  to 
the  SSE  Community  a 
Guidebook  to  Using  Trusted 
Suppliers 

►  Mr.  Jim  Gobes,  Intrinsix 
Corp. 

Track  2 

DC 

LJJ 

—I 

Education  & 
Training 

4C2 

19811 

Version  1 .0  of  the  New 
INCOSE  Competency 
Framework 

►  Mr.  Don  Gelosh 

19515 

A  Proposed  Engineering 
Training  Framework  and 
Competency  Methodology 

►  Dr.  Eric  Dane, 

BAE  Systems 

19695 

Educating  Engineers  or 
Training  Technicians 

►  Mr.  Zane  Scott, 

Vitech  Corporation 

19734 

Solving  Cybersecurity 

Skills  Shortage  With 
Apprenticeships  & 
Certifications  -  A  Case 

Study 

►  Mr.  Girish  Seshagiri, 

Ishpi  Information 
Technologies,  Inc. 

Track  3 

Von  Sternberg 

Engineered 
Resilient  Systems 

4C3 

19783 

The  Language  of 

Complexity:  Ontology 
in  Systems  Design  and 
Engineering 

►  Mr.  Abe  Wu, 

Raytheon  Missiles 

19846 

Physics  and  Model  Based 
Aerodynamic  Design  and 
Analysis  at  GA 

►  Mr.  Pritesh  Mody, 

General  Atomics 

Aeronautical  Systems,  Inc. 

20050 

Automation  and  Integration 
for  Complex  System  Design 

►  Mr.  Scott  Radon,  Phoenix 
Integration 

19825 

Application  of  CREATE  Tools 
for  High  Fidelity  Design 

Space  Exploration 

►  Mr.  Antonio  De  La 

Garza,  Lockheed  Martin 
Aeronautics  Company 

Track  4 

Gibson 

Program 

Management 

4C4 

19751 

A  Capability  Value  Frontier 
in  Support  of  Acquisition 
Approaches  to  Enable 

Military  Effectiveness 

►  Dr.  Marilyn  Gaska, 
Lockheed  Martin 

Corporation 

19782 

Technical  Data  Package  and 
Intellectual  Property  Rights 

►  Mr.  William  Decker, 

Defense  Acquisition 

University 

19827 

Policy  Engineering:  Applying 
Systems  Engineering  to 
Develop  Better  Policies 

►  Dr.  Steven  Dam, 

SPEC  Innovations 

Track  5 

Seller 

Environment, 
Safety  & 
Occupational 
Health 

4C5 

19714 

DoD’s  REACH  Strategy  and 
its  Impact  to  Acquisition  and 
Sustainment 

►  Dr.  Patricia  Underwood, 
OASD(EI&E) 

19705 

Environmental  Liabilities  for 
DoD  Weapons  Systems 

►  Ms.  Patricia  Huheey, 
OASD(EI&E) 

19810 

Environmental  Life  Cycle 
Assessment  of  Commercial 
Transportation  Activities 

►  Ms.  Sheila  Neumann, 
University  of  Texas  at 
Arlington 

19699 

Life  Cycle  Assessment:  A 

Tool  for  Protecting  Defense 
Assets 

►  Dr.  Kelly  Scanlon, 
OASD(EI&E) 

CD 

o 

< 

A 

Korman 

Architecture 

4C6 

19748 

Advancing  U.S.  Marine 

Corps  Warehouse 
Management  Operations 
Through  System 

Architecture  and  Analysis 

►  Mr.  Christopher 

Melkonian, 

Marine  Corps  Systems 
Command 

19828 

From  Architecture  to 
Operations  -  Using  Your 
Architecture  Work  in 
Operations 

►  Dr.  Steven  Dam, 

SPEC  Innovations 
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SYSTEMS  ENGINEERING  CONFERENCE 


Thursday,  October  26  -  Continued 

2:40pm  -  3:15pm  Networking  Break 


3:15pm  -  3:40pm 

3:40pm  -  4:05pm 

4:05pm  -  4:30pm 

Track  1 

Singleton 

System  Security 
Engineering 

4D1 

19864 

Field  Programmable  Gate  Array 
(FPGA)  Assurance 

►  Mr.  Ray  Shanahan,  Department  of 
Defense 

19891 

Using  Cyber  Resiliency  Frameworks 
to  Engineer  and  Manage  IT  Services 

►  Dr.  Subash  Katie, 

The  MITRE  Corporation 

19863 

Survey  of  Cyber  Security  Framework 
across  Industries 

►  Mr.  Ambrose  Kam, 

Lockheed  Martin  Corporation 

Track  2 

Miller 

Education  & 
Training 

4D2 

19756 

Teaching  Executable  Model-Based 
Engineering  (MBE):  Best  Practices 

►  Mr.  Matthew  Cotter, 

The  MITRE  Corporation 

19760 

The  Systems  of  Systems  (SoS) 

Primer:  A  Guide  to  SoS  for  all 

Expertise  Levels 

►  Ms.  Laura  AntuI, 

The  MITRE  Corporation 

19865 

Breaking  Out:  Systems  Engineering 

To  Go 

►  Mr.  Zane  Scott, 

Vitech  Corporation 

Track  3 

Von  Sternberg 

Engineered 
Resilient  Systems 

4D3 

19712 

Implementation  of  Clustering  Analysis 
in  Engineered  Resilient  Systems  Tools 
for  Enhanced  Trade  Space  Exploration 
of  Military  Ground  Vehicles 

►  Mr.  Andrew  Pokoyoway, 

TARDEC 

19818 

Tradespace  Analysis  and  Exploration 
incorporating  Reliability,  Availability, 
Maintainability,  and  Cost 

►  Dr.  Lance  Fiondella, 

University  of  Massachusetts 

19741 

Security  at  Design  Time:  Addressing 
Resilience  in  Mission  Critical  Cyber- 
Physical  Systems 

►  Mr.  Thomas  McDermott, 

Georgia  Tech  Research  Institute 

Track  4 

Gibson 

Program 

Management 

4D4 

19847 

Proactively  Managing  Supplier 
Relationships  for  an  Integrated 

Product  Development  Program 

►  Ms.  Beth  Layman, 

Layman  &  Layman 

19932 

Improving  Efficiency  in  Assembly, 
Integration  and  Test  (AI&T) 

►  Mr.  Jeff  Juranek, 

The  Aerospace  Corporation 

19842 

“Other  Transactions”  -  An  Alternative 
to  Business  as  Usual 

►  Mr.  Richard  Dunn, 

Strategic  Inst  for  Innovation  in  Govt 
Contracting 

Track  5 

Sellier 

Environment, 
Safety  & 
Occupational 
Health 

4D5 

19766 

ESOH  Management  in  Agile  and 

Rapid  Acquisitions  Using  Digital 
Engineering 

►  Mr.  Sherman  Forbes, 

SAE/AQRE 

Track  6 

Korman 

Enterprise  Health 
Management 

4D6 

19523 

Mission-Based  Forecasting  for  the 
Sustainment  Enterprise 

►  Col  Greg  Parlier,  USA  (Ret.), 

GH  Parlier  Consulting 
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SYSTEMS  ENGINEERING  CONFERENCE 


Thursday,  October  26  -  Continued 


4:30pm  -  4:55pm 

4:55pm  -  5:20pm 

Track  1 

Singleton 

System  Security 
Engineering 

19722 

The  Systems  Challenges  of 
Cybersecurity 
►  Mr.  Jeffery  Zili, 

Vitech 

19895 

Modeling  Cyber  Security 

►  Mr.  Ambrose  Kam, 

Lockheed  Martin  Corporation 

4D1 

Track  2 

cc 

LU 

-I 

_l 

Education  & 
Training 

19914 

Bridging  the  Gap  to  MBSE 

►  Mr.  James  Baker, 

Sparx  Systems 

19719 

Introducing  Cyber  Resiliency  Concerns 
Into  Engineering  Education 

►  Mr.  Thomas  McDermott, 

Georgia  Tech  Research  Institute 

4D2 

Track  3 

Von  Sternberg 

Engineered 
Resilient  Systems 

4D3 

19781 

Additive  Manufacturing  -  Challenges... 
Program  Manager 

►  Mr.  William  Decker, 

DAU  Huntsville 

20051 

Model-Based  Engineering: 

Opportunities,  Risks,  and  Best 

Practices 

►  Dr.  Marc  Halpern, 

Gartner,  Inc. 

5:20pm  Adjourn  Conference 
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SILVER  SPONSORS 


At  IBM  Research,  we  invent  things  that  change  the  world,  We  are  pioneering  promising  and  disruptive  technologies  that  will  transform  industries 
and  society,  including  the  future  ot  Al,  blockchain  and  quantum  computing. 

We  are  driven  to  discover.  We  are  home  to  more  than  3,000  researchers  in  1 2  labs  located  across  six  continents,  Scientists  from  IBM  Research 
have  produced  six  Nobel  Laureates,  1 0  U.S,  National  Medals  of  Technology,  five  U.S,  Nafional  Medals  of  Science,  6  Turing  Awards,  1 9  inducfees 
in  fhe  National  Academy  ot  Sciences  and  20  inductees  into  the  U.S.  National  Inventors  Hall  of  Fame. 

Our  teams  are  pushing  the  boundaries  ot  science  to  uncover  tomorrow’s  breakthroughs  for  national  security,  economic  growth  and  jobs.  We  are 
especially  focused  on  microelectronics  as  a  national  critical  resource.  The  semiconductor  industry  is  a  foundational  industry  for  modern  sociefy. 
Semiconductors  enable  all  electronics;  they  are  at  the  base  of  the  electronics  food  chain  and  make  digital  life  -  every  electronics  system  in  the 
world  -  possible.  Technological  leadership  in  semiconductor  research,  development,  design  and  manufacturing  is  vital  for  economic  growfh  and 
especially  for  nafional  securify. 


LOCKHEED 

"Headquartered  in  Bethesda,  Maryland,  Lockheed  Martin  is  a  global  security  and  aerospace  company  that  employs  approximately  97,000 
people  worldwide  and  is  principally  engaged  in  the  research,  design,  development,  manufacture,  integration  and  sustainment  of  advanced 
technology  systems,  products  and  services." 


RayHieon 

Raytheon  Company  is  a  technology  and  innovation  leader  specializing  in  defense,  security  and  civil  markets  throughout  the  world.  With  a  history 
ot  innovation  spanning  more  than  90  years,  Raytheon  provides  state-of-the-art  electronics,  mission  systems  integration  and  other  capabilities 
in  the  areas  of  sensing;  effecfs;  and  command,  confrol,  communications  and  intelligence  systems;  as  well  as  a  broad  range  of  mission  supporf 
services. 
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Outpacing  the  Competition:  A  Systems  Engineering  Chaiienge 
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Presented  To: 

NDIA  Systems  Engineering  Conference 

Presented  By: 


VADM  Paui  Groskiags,  Commander,  NAVAiR 


Day  in  the  life  of  an  SE  dealing  with  PMs 
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I  Framing  the  Challenge 


Life  Has  Been  Good! 


Sea  Lanes  Remain  the  Lifeblood  of  Our  Economy 


Competition  is  Back 
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Changing  Environment 
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Changing  Environment 


Short  Range 
Ballistic  Missiles 


AAAAAAAAA 

AAAAAAAAl 

11111111 J 


Advanced  Fighter/  Strike 
Aircraft  (3'^^/4*'^  Generation) 


Advanced  ASCM  Capable 
Surface  Combatants  (DD/FF) 


Aircraft  Carrier 


PRC  Military  Spending  (Official  Figures) 


120 


Spending  in  Billions  (US) 


1999  2001  2003  2005  2007  2009  2011  2013 


2014  Budget  Increased  12.2% 


1999 
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Advanced  Attack 
Submarines 


2015 


USN  and  PLA(N)  Capability  Fielding  Trends 
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y^int  Strike  Fighter  (F-3^^ 
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USN  Warfighting  Advantage  has  Steadily  Eroded 
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CNO’s  Challenges  to  all  Flag/SES 


5  Key  Points 


Must  be  competitive  ^  Existentiai  Threat  ^  No  #2 
Think  Strategicaiiy  ^  Criticai  Thinking 


Going  Digitai 

Outcome  /  Product  Oriented  ^  Vice  Process 
Sense  of  Urgency  ^  Shouid  be  Uncomfortabie 


‘If  It’s  Not  Making  the  Fleet  More  Lethal  - 

Stop  Doing  It!” 
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NAVAIR  Response 


strategic  Initiatives  -  Focus  on  Speed 


•  Capabilities  Based  Acquisition  -  Rapid  delivery  of  integrated  capabilities 

•  Sustainment  Vision  2020  -  Predictive,  integrated  sustainment  operations 

•  Digital  Business  Operations  -  Integrated  business  systems  "apps"  at  the  desktop 


Accelerating  delivery  of  fully  integrated  capabilities  which  are  designed, 
developed,  and  sustained  in  a  Model  Based  Digital  Environment 

IS  a  Systems  Engineering  challenge 


11 


Distribution  Statement  A: 
Approved  for  public  release;  distribution  unlimited  PR#  201 7-928 


Capabilities  Based  Acquisition 

Digital  Thread  Enables  Rapid  Delivery  of  Integrated  Capabilities 
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Integrated  Warfare  Analysis 
establishes  CONEMPS, 
Effects-Chains,  required  attributes 


SoS  Model 


CONEMPS  and  Effects  Chains  are 
modeled  at  the  System  of  Systems 
(SoS)  level 


System  models  form  “Constructive” 
basis  for  LVC  M&S  environment 


Systems  are  developed  in  a 
Model-Based  environment 
(SE  Transformation) 


I  El 


Digital  Linkage 


Enabling  Capabiiities-Based  T&E 


Constructive  Virtual  Live 
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SET  Framework 


Elimination  of  paper  CDRL  artifacts  and 
large-scaie  design  reviews 


Continuous  insight/oversight  via  digitai 
coliaborative  environment  and  interaction  with  the 

Singie  Source  of  Truth 
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Multi-Disciplinary  Analysis  &  Optimization 
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Execution  Framework 
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SET  Research  Team 
(Blackburn) 
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Surrogate  System  Experiment 


•  Simulate  Execution  of  SET  Framework 

•  Use  UAV  scenario  developed  in 
SERC  models 

-  Combine  SysML  models  already  in 
development  -  requirements, 
with  functional  and  logical  views 

-  Use  MDAO  of  parametrics  for  some  KPPs 

•  Consider  NATO  example 

•  Characterize  objectives  and  thresholds 

-  Create  a  model-based  contract  simulating  RFP  /  SOW 

•  Use  commercial  organization  to  simulate  industry  organization 

-  Refinement  of  SysML  models  to  reflect  corrections  /  innovations  with  physical  allocation 
views 

-  Integrate  with  multi-physics-based  Initial  Balanced  Design 

-  Simulate  continuous  virtual  reviews  and  derive  new  objective  measures  for  assessing 
maturing  design 

•  Simulate  source  selection  based  on  dynamic  models  and  simulations 


Elimination  of  paper  CDRL  artifacts  and 
large-scale  design  reviews 
Continuous  insight/oversight  via  digital 
collaborative  environment  and 
interaction  with  the  Single  Source  of 
Truth 


oyvivin  opev  m  j 

_  » e»h».K.- 

Mochanical  Dosign  Modols 

it 

Electrical  Design  Models 

Ifel 

SoftMare  Design  Models 

t  .mB  ' 

Testing  Methods  &  Models 

Analysis  Tools 

MDAOVSET-BASED  DESIGN 


*  Muki-Discipfinary  Analysis  &  Optimization 
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Industry-Government  Partnership 


•  SET  applies  to  both  Government  and  Industry 

•  Government  must  reassess  its  role  in  the  acquisition 
process  and  the  methods  for  executing  that  role 

1 .  Criteria  for  gov’t  involvement  /  oversight  (not  every  decision) 

2.  If  involved,  must  be  on  developer’s  timeline 

3.  Must  bring  value  to  the  decision  -  not  just  positional  authority 

•  Industry  must  fully  leverage  advances  in  HPC-enabled 
models  and  participate  in  establishing  a  collaborative, 
integrated  digital  environment  which 
enables  continuous  interaction 


16 


Distribution  Statement  A: 
Approved  for  public  release;  distribution  unlimited  PR#  201 7-928 


For  More  Information,  Contact: 


Mr.  Dave  Cohen,  Director  Systems  Engineering 

(301)757-5542 

david.cohen@navv.mii 


-  ''Wvvv^vW 


Capabilities  Based  Acquisition 


Integrated 
Warfare  Analysis 

Fleet  operators  work 
with  engineering  to  develop 
concepts  of  employment 
(CONEMPS),  effects-chains 
&  required  attributes 


CONEMPS  and 
effects-chains  are 
modeled  at  the 
System  of  Systems 
(SoS)  level 


System  Model 

Systems  are  developed 
in  a  model-based  integrated 
digital  environment 
(SE  Transformation) 


Industry  Models 
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Results  in  optimized 
capability  development 
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Enabling 
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System  models  form 
Constructive" 
basis  for  LVC 
M&S  environment 


Virtual 

Hardware 
in  the  loop 

Integration 
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Installed 
Systems 
Test  Facility 
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Live 

Flight  Test 
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Required  for  Fleet  Integrated  Ops 


Integrated  Digital  Environment  acceierates  deiivery  of  operationaliy  relevant  capabilities 
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Sustainment  Vision  2020  -  What  it  Looks  Like 


RAW  DATA 


APPLICATIONS 
/  TOOLS 


ANALYSIS 


FLEET  DECISIONS 
FLEET  SUPPORT 


STATUS-TRENDS-PREDICTIONS 

IS 


Naval  Aviation  Sustainment  Center 


Universal  Information 
Faster  Decision  Making 
Predictive  Sustainment  Planning 
Reduced  Cost 
Increased  Readiness 


as  Optimization  and 
W  Prioritization  of 

Resources  to  Meet 
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Maintenance  Planning 
Supply  Support 
Workforce 
Facilities 
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Data 


Digital  Transformation:  Business  Operations 


I 


Today:  Monoliths  in  Silos 


Tomorrow:  The  Composable  Business 


I 


Applications  built  in  silos 
Data  duplicated  in  tools  causing  manual  re-entry 
Data  locked  in  tools  preventing  ease  of  re-use 
Application  changes  slow  and  costly 

Functionality  duplicated  across  tools  causing  inconsistencies  and 
difficulty  in  coordinating  business  process  changes 
Can’t  tailor  to  support  unique  process  across  business  units 
Similar  functionality  reinvented  over  and  over  again  at  great  cost 


Infrastructure  enables  tailored  applications  while  maintaining  consistent  core 
business  rules  and  data 

Applications  “composed”  from  reusable  services 

Consistency  of  Data  and  Business  Rules  across  Business  Operations 

Agility  in  supporting  rapid  Business  Process  changes 

Lightweight  services  with  short  development  lifecycle 

Individual  Services  “owned”  by  Authoritative  Competency 
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USN  vs  PLA(N)  Capability  Fielding 


We’re  Being  Out-Sticked 


USN  Warfighting  Advantage  Against  PLA(N)  has  Steadily  Eroded 
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Outline 


•  Overview 

-  What  the  DMSRA  is  and  isn’t 

-  GoalsA/ision/Motivation 

-  Composable  simulation  architecture 

•  Challenges 

-  Architectural  and  engineering 

-  Enterprise-wide  interoperability  and  reuse 

•  Best  practices  (patterns) 

-  Identified 

-  Planned  additions 

•  Conclusions 

— u - ^inco 


New  slide 
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Overview 


•  The  DMSRA  is  NOT  a  solution  architecture. 

•  It  establishes  a  vision  for  Defense  M&S: 

-  that  leverages  emerging  technologies,  and  enterprise  services; 

-  to  promote  reuse  and  interoperability. 

•  The  DMSRA  provides  broadly  applicable 
guidance. 

-  It  captures  principles,  standards,  and  best  practices  for  simulation 
architects  and  engineers  to  align  on  the  vision. 

-  It  is  not  mandatory. 


^iirsco 


New  slide 
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DMSRA  Vision 


A  robust  modeling  and  simulation  (M&S)  capability  that  supports  a  fuil 
spectrum  of  DoD  activities  and  operations,  delivered  to  the  point  of  need, 
within  current  fiscal  constraints,  managing  schedules  and  risk  enabled  by 

agile  composition. 


•  Models  and  simulations  that: 

-  Are  modular  -  decomposed  into  loosely  coupled  reusable 
components; 

-  Execute  in  the  cloud  {where  practical)  -  hosted  in  thft^tCMl,  and  are 
capable  of  taking  advantage  of  cloud  characteristics  ii^f  as  remote 
access  and  scalability; 

-  Adhere  to  enterprise-wide  composability  standards-  follow 
standards  that  facilitate  the  reusability  of  components  across 
programs  and  Components. 


^JfSCO 


This  slide  was  cleared  in  DOPSR  Case  #17-5-2193,  slide  3: 

DMSRA  Motivation: 

The  Federal  Government  and  DoD  have  IT  strategies  promoting  the  adoption  of  cloud  computing 

•Initiatives  are  already  under  way  such  as  the  the  Joint  Information  Environment  (JIE)  which  will 
change  how  the  department  does  IT  business 
•The  M&S  community  must  prepare  for  the  coming  changes 
Technologies  such  as  cloud  computing  and  service-oriented  architecture  (SOA)  can  provide  significant 
opportunities 

•To  improve  accessibility  and  agility 
•While  reducing  operating  and  maintenance  costs 
These  technologies  inherently  promote  reusability 

•By  coordinating  their  implementation  across  the  department  we  can 
Reduce  development  time  and  cost 
Increase  simulation  accuracy  through  component  reuse 
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[Note:  This  figure  was  cleared  by  DOPSR  on  June  6,  2016  (Case  #16-5-2052,  page  5  of 
12)] 

DMSRA  Structure: 

Strategic  Purpose 

Goals  and  objectives  of  the  DMSRA;  specific  purpose  of  and  the 
problenn(s)  to  be  addressed  by  the  DMSRA. 

Principles 

High-level  foundational  statements  of  stakeholders,  rules,  and 
values  that  drive  technical  positions  and  patterns. 

Technical  Positions 

Technical  guidance  and  standards,  based  on  specified  principles  that 
need  to  be  followed  and  implemented  as  part  of  the  solution. 
Patterns  (Templates) 

Generalized  architecture  representations  (viewpoints, 
graphical/textual  models,  diagrams,  etc.)  that  show  relationships 
between  elements  and  artifacts  specified  by  the  technical  positions 
and  encourage  adherence  to  common  standards,  specifications  and 
patterns. 

Vocabulary 
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[Note:  This  figure  was  cleared  by  DOPSR  on  June  6,  2016  (Case  #16-5-2052,  page  9  of 

12)] 

Notes  from:  table  cleared  by  DOPSR  on  June  6,  2016  (Case  #16-5-2052,  page  9  of  12): 
Advantages: 

•  Models  are  coded  once,  reducing  development  time  and  cost 

•  Easy  to  replace  models  with  newer  versions  that  use  the  same  interface 

•  Smaller  simulations  should  lead  to  easier  use  and  reduced  maintenance  costs 

•  Conducive  to  cloud  computing  infrastructure 
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Architectural  and  Engineering 
Challenges 


•  Managing  a  hybrid  architecture  that  maintains 
interoperability  with  legacy  systems 

•  Decomposition  of  legacy  systems  into  reusable 
components 

•  Development  of  standards  to  facilitate  composability 
of  models 

-  Common  conceptual  model/framework  for  assembling 
components 

-  Verification  and  Validation  of  composed  simulations 


^JfSCO 


Includes  content  from  table  cleared  by  DOPSR  on  June  6,  2016  (Case  #16-5-2052,  page 
9  of  12)] 
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Unique  M&S  Challenges  to 
Modular,  Open  System  Approach 


□  Account  Number 
Z1  ^uthenticatiDii 


□  Amount 

□  Carrent^ 

□  TijnieStarnp 


OAN*’' 

^  r  AcccMjnt  E>aTab9Ei« 


The  bank  keeps  the  definitive  record 
of  the  amount  of  money  in  an 
account 


□  OtSilve/'Coorclin^lj 

□  Target  CoondjnaEej 


□  True/Faise 


Terr^ 


-.'rBTftfri 
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The  terrain  database  Is  a 
representation  of  the  terrain  based 
on  a  set  of  simplifying  assumptions; 

those  assumptions  affect  the 
suitability  and  accuracy  of  the  data 


#>iraco 


[Note:  This  slide  was  cleared  by  DOPSR  (Case  #17-5-2610,  page  12)] 
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Enterprise-wide  Interoperability 
and  Reuse  Challenges 


*  Implementing  governance  structures  that  enable  and 
encourage  modular,  open-systems  approaches 

•  Facilitating  trust  between  simulation  developers, 
dependent  upon  other  model  and  simulation 
developers  who  may  not  be  in  their  program  chain. 

-  This  will  require  simulation  program  managers  to  accept  some  risk 

-  It  will  also  require  adoption  of  common  conceptual  model  (s)  or 
frameworks 


^JfSCO 


Includes  content  from  table  cleared  by  DOPSR  on  June  6,  2016  (Case  #16-5-2052,  page 
9  of  12)] 


9 


How  the  DMSRA  is  Addressing  the 
Challenges 

•  Collaborative  approach 

•  Leverage  existing  investments 

•  Develop  patterns  that  capture  best  practices, 
and  gaps  in  standards,  technology  and  practice 


New  slide 
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Collaborative  Approach 


M&S  COI  Architecture  Working  Group  (AWG) 

-  36  briefings  on  architecture /framework  initiatives 


-  Includes  briefings  from  all  4  Services.  MDA,  Joint  Staff,  and  NATO 

Training 

T&E 

Acquisition 

❖  Expenmentahon 

❖  Analysis 

Online  collaboration  ^ 
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Emphasizes  the  dynamic  and  collaborative  nature  of  the  DMSRA 
Makes  the  revision  process  more  transparent 
Makes  it  easier  to  contribute  to  the  DMSRA 
Makes  contributions  immediately  available  and  easier  to  find 
https;//www. milsuite.mil/book/groups/dmsra  (DoD  CAC  only) 


-MSM 


[Note:  Portions  of  this  slide  were  cleared  by  DOPSR  on  for  presentation  at  l/ITSEC 
2016.  (case  #  16-S-2705,  slide  6.)] 

[Note:  The  number  of  briefings  will  need  to  be  updated  after  the  SIW  to  reflect  the 
briefings  given  at  the  special  session.] 
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Leveraging  Existing  investments 


•  The  DMSRA  effort  builds  on  the  Live,  Virtuai, 
Constructive  Architecture  Roadmap  (LVCAR) 
principies: 

•  Do  no  harm 


•  Interoperability  is  not  free 

•  Start  with  small  steps 

•  Provide  central  management 


•  Other  investments  and  resources  leveraged: 

•  Defense  M&S  Glossary 

•  Verification,  Validation,  and  Accreditation  (WSA)  Recommended 
Practices  Guide 

•  DoD  and  NATO  standards  references  and  tools 

•  Services'  architecture{s)  artifacts  and  practices 


12 


[Note:  This  slide  was  cleared  by  DOPSR  (case  #  16-S-2705,  slide  5)] 
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Patterns: 

Best  Practices  and  Gaps 


•  Extensibility  via  Patterns 

-  The  base  document  and  initial  patterns  were  not  sufficiently  comprehensive  to  meet 
the  DMSRA  vision 

-  Led  to  the  use  of  modular  patterns  to  extend  and  evolve  the  DMSRA  with  new 
technologies  and  associated  best  practices. 

*  DMSRA  Pattern  Outline: 

-  Pattern  overview:  Frames  topic  with  definitions,  technology  descnptlon, 
and  relevance  to  the  DMSRA 

-  Mapping  from  Capabilities,  and  Principles  and  Rules:  aligns  capability 
with  DMSRA  principles 

—  Pattern:  Provides  a  series  of  questions  the  user  should  ask  in  the  process  of 
deciding  whether  to  apply  the  te  c  h  no  logy/cap  ability.  Documents  guidance  and  best 
practices  for  answering  the  questions  in  context  based  on  inputs  from  the  AWG, 

-  Technical  Positions:  Identifies  applicable  standards,  including  DoD 
adoption  status;  and  standardization  gaps 

-  References 

— rz - ^irsco 


[Note:  Portions  of  this  slide  were  cleared  by  DOPSR  (case  #  17-S-2610,  slide  15)] 
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Current  Patterns  Findings  (1  of  2) 


Cloud  migration 

-  Lowpi,p^all  costs  to  the  consumer,  because  of  efficiencies 
obt^^!^  pooling  much  of  the  computing  hardware  and  software; 

-  IT  and  increased  flexibility  because  there  is  no  upfront 

investment  in  infrastructure  required  by  the  end  user 


•  Service-oriented  architecture 


-  The  Department  of  Defense  (DoD)  Chief  Information  Officer  (CIO) 

has  directed  the  DSi^-^ieverage  commercial  SOA  technologies  to 
reduce  costs  and  flexibility. 

-  This  pattern  aids  ttie  user  to  determine  the  suitability  of  an 
organizational  capability  for  migration  to  a  SOA  from  technical, 
programmatic,  and  domain  perspectives. 


<&inco 


New  slide 


14 


Current  Patterns  Findings  (2  of  2) 


•  Decomposition  of  simulations  into  modular 
components 

-  Although  much  has  been  written  about  modular  simulation,  there  is  a 
gap  for  M&S-specific  standard  practices  for  decomposition. 

•  Verification  and  validation  of  modular 
components 

-  Cloud  computing  considerations;  The  hardware  andJQjp>eri^ing 
system  the  simulation  is  hosted  on  are  out  of  the  con^tofthe  user 
and  may  be  altered  from  the  configuration  used  durii^^^ation 
without  the  user’s  knowledge, 

-  V&V  of  composed  simulations:  ^^^|sition  of  validated  component 
models  does  not  ensure  a  valid^^^^sed  simulation.  This  is  a 
known  gap  in  standards  and  practice. 


New  slide 


15 


Way  Ahead 


Continue  collaborative  approach  to  capturing  best 
practices  in  patterns,  including  the  following  topics: 

•  Accommodating  occasional  /  sporadic  connectivity 

•  Cross  domain  solutions 

•  Distributed  simulation  and  federation  engineering 

•  Data 

•  Assessing  the  feasibility  of  remote  execution 

•  Gaming  architectures 

Continue  to  leverage  DoD  enterprise  architecture  and  IT 
capabilities  and  practices: 

•  Cloud  computing 

•  MOSA  and  SOA  practices  and  standards 

-t; - MSCO 


[Note:  Portions  of  this  slide  were  cleared  by  DOPSR  (case  #  17-S-2610,  slide  16)] 
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Agenda 

•  Some  Givens 

•  Safety  versus  Security 

•  General  Static  Analysis:  Dealing  with  false  positives  and  false 
negatives 

•  Targeted  Static  Analysis:  Proving  specific  properties  and  assertions 

•  Coordinating  the  Efforts 

•  Conclusion 


NOTE:  Blue  highlighting  in  this  presentation  is  for  emphasis. 
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The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Some  Givens 


[C]ybersecurity  applies  to  weapons  systems  . . .  [and]  is  a  critical 
priority  for  the  DoD. . .  incorporate  code  reviews  and  architecture 
reviews  against  incremental  builds  to  reduce  vulnerabilities  in  any 
custom  software,  including  via  automated  scanning  tools  (e.g., 
static  analysis). 

[The  DoD  Program  Manager’s  Guidebook  for  Integrating  the 
Cybersecurity  Risk  Management  Framework  (RMF)  into  the  System 
Acquisition  Lifecycle,  September  2015] 


DoD  will  continue  to  assess  Defense  Federal  Acquisition 
Regulation  Supplement  (DFARS)  rules  ...  to  ensure  they  mature  . 
. .  in  a  manner  consistent  with  known  standards  for  protecting  data 
from  cyber  adversaries,  to  include  standards  ...  by  the  National 
Institute  of  Standards  and  Technology  (NIST). 

[The  Department  of  Defense  Cyber  Strategy,  April  2015] 


The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

More  Givens 

Source  code  should  be  periodically  reviewed  using  automated  tools  or 
manuai  spot  check  for  common  programming  errors  ...  as  part  of  the 
software  deveiopment  QA  process. 

[NIST  Special  Publication  800-64  revision  2,  Security  Considerations  in  the 
System  Development  Life  Cycle,  October  2008] 


The  Program  Manager  will  integrate  ESOH  risk  management  into  the 
overaii  systems  engineering  process  for  ali  engineering  activities 
throughout  the  system’s  life  cycle.  . .  The  Program  Manager  will  use  the 
methodology  in  MiL-STD-882E. 

[DoD  Instruction  5000.02,  “Operation  of  the  Defense  Acquisition  System,” 
January  7,  2015] 


Level  of  Rigor  Tasks  [for  Software  Criticaiity  index  (SwCi)  1 /highest] . . . 
Program  shall  perform  analysis  of  requirements,  architecture,  design, 
and  code;  and  conduct  in-depth  safety-specific  testing. 

[MIL-STD-882E,  “DoD  Standard  Practice  for  System  Safety,”  May  11,  2012] 


The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Software  Safety  versus  Software  Security 


Software  Securit' 
Focus 

External  interface 
vulnerabilities 
Vulnerabilities  to 
malicious  intent 


Software  Safety 
Focus 

Internal  data  corruption 
vulnerabilities 
Time  critical  latency 
issues 

Vulnerabilities  to 
unintended  mistakes  in 
design  or  implementation 


There  is  some  overlap,  but  the  priorities 
and  focus  are  different. 


The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Software  Safety  versus  Software  Security 


Software  Safety 
Focus 

Race  conditions  with 
safety-critical  data 
Latency  issues  with 
safety-critical  response 
or  data  update 
Inadequate  or  erroneous 
feedback  to  an  operator 


Software  Securit] 
Focus 

Missing/incorrect 
authentication  or 
authorization 
Injection  of  malicious 
data  or  scripts 
Uncontrolled  data  or 
buffer  overflow 


The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Software  Safety  versus  Software  Security 


Security  issues  tend  to  be  at  the  external  interfaces 

of  a  software  application. 


The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Software  Safety  versus  Software  Security 


Software 

application 


Safety  issues  tend  to  be  in  the  core  system 
functionality  of  a  software  application. 


General  Static  Analysis: 

Dealing  with  false  positives  and 
false  negatives 


The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

General  Static  Analysis 

►  General  static  source  code  analysis 
-  Flagging  programming  errors 

•  MITRE’S  Common  Weakness  Enumeration  (CWE) 

•  False  positives  and  false  negatives 


►  Targeted  static  analysis 

-  Proving  targeted  assertions 

-  Counter  examples 

-  Program  slicing 


The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

General  Static  Source  Code  Analysis 

►  Flagging  programming  errors 

-  MITRE’S  Common  Weakness  Enumeration  (CWE) 

-  Security  CWE’s 

•  Open  Web  Application  Security  Project  (OWASP)  Top  10  CWE’s 

o  Injection  /  Broken  Authentication  /  Cross-site  Scripting  / 

Insecure  Direct  Object  References  /  Security  Misconfiguration  / 
etc. 

-  Safety  CWE’s 

•  Data  corruption  CWE’s 

o  Shared  resource  race  condition  /  Buffer  Overflow  /  Improper 
Validation  of  an  Array  Index  /  Pointer  Issues  /  Incorrect  Type 
Conversion  /  etc. 


Booz  Allen  Hamilton 
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The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Safety  Critical  Data  ‘Corruption’ 

A  correctly  implemented  algorithm  operating  on  corrupted  or  stale 
safety-critical  data  can  have  unintended  catastrophic  results. 

Some  sources  of  corrupted  data: 

•  Noise  In  digital  message  transmission 

•  Physical  events/upsets  during  data  storage 

•  Multi-threaded  shared  data 

•  Shared  data  between  ‘main’  and  Interrupt  Service  Routines 

•  Caching  of  data 

•  Loss  of  transient  status  data  in  failover  or  ‘recovery’ 


Booz  Allen  Hamilton 
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Ihe  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

General  Static  Code  Analysis 


Safety 

Vulnerabilities 


V,.  Static  Code  Analysis  Tool  Coverage 

✓ 

/ 

/ 

The  tools  cover  many,  but  not  all,  vulnerabilities. 

' There  are  false  positives  and  faise  negatives  with  every  tooi. 


Booz  Allen  Hamilton 
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The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

The  Opportunity  for  Software  Safety 

►  Many  of  the  programming  errors  detected  by  software  static  analysis 
toois  used  for  cybersecurity  have  potentiai  safety-critical  impacts: 

-  Muiti-threaded  race  conditions 

-  Mishandiing  of  pointers 

-  incorrect  casting  (data  type  conversion) 

-  Buffer  overfiow 

►  Providing  access  to  generai  static  analysis  toois  aiready  being  used 
for  cybersecurity  couid  greatly  assist  those  responsible  for  software 
safety  design  and  code  analysis. 

-  Need  communication  and  coordination  of  effort  between  those 
responsible  for  security  and  those  responsibie  for  system  safety 


Booz  Allen  Hamilton 
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The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Static  analysis  tools  are  already  in  use  for  safety 

►  Food  and  Drug  Administration  (FDA): 

. . .  static  analysis  examines  the  code  exhaustively  for  certain  kinds 
of  insidious  errors  that  are  hard  for  human  reviewers  to  detect. 

[http://vvww.fda.gov/MedicalDevices/ProductsandMedicalProcedures/GeneralHospitalDevicesandSupplies/l 

nfusionPumps/ucm20251 1  .htnn#staticAnalysis] 

►  Federal  Aviation  Administration  (FAA): 

A  combination  of  both  static  and  dynamic  analyses  should  be 
specified  by  the  applicant/developer  and  applied  to  the  software. 

[Certification  Authorities  Software  Team  (CAST)  Position  Paper  CAST-9,  January  2002] 

►  Motor  Industry  Software  Reliability  Association  (MISRA): 

Compliance  with  MISRA  C/C++  coding  standards  for  safety-critical 
software  is  checked  by  many  static  analysis  tools. 


Booz  Allen  Hamilton 
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The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Some  General  Static  Source  Code  Analysis  Tools 

►  Flagging  programming  errors 

•  Grammatech’s  CodeSonar 

•  Coverity’s  Code  Advisor 

•  IBM’s  AppScan 

•  Clang  Static  Analyzer 

•  CppCheck 

•  Parasoft’s  Static  Analysis  Engine 

•  Redlizard’s  Goanna 

•  Checkmarx’s  CxSAST 

•  Fasoo’s  Sparrow 


Booz  Allen  Hamilton 
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Targeted  Static  Analysis: 

Proving  specific  properties  and 

assertions 


The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Targeted  Static  Analysis 


►  Targeted  static  analysis 

-  Proving  targeted  assertions 

-  Counter  examples 

-  Program  slicing 


Booz  Allen  Hamilton 
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The  Givens 

Safety  vs  Security  | 

I  Generai  Anaiysis  | 

I  Targeted  Anaiysis  | 

I  Coordination 

Conciusion 

Targeted  Static  Analysis 
Abstract  Interpretation/Model  Checking 


for  any  possible  execution  sequence  (absence  of  spec/7/c  vulnerabilities). 


Booz  Allen  Hamilton 
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The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Soundness  vs.  Completeness 

“[T]he  essence  of  [abstract]  static  analysis  is  to  efficiently  compute 
approximate  but  sound  guarantees:  guarantees  that  are  not  misleading. 
. . .  Due  to  the  undecidability  of  static  analysis  problems,  devising  a 
procedure  that  does  not  produce  spurious  warnings  and  does  not  miss 
bugs  is  not  possible.” 

[“A  Survey  of  Automated  Techniques  for  Formal  Software  Verification”  D’Silva,  et  al.  IEEE 
TRANSACTIONS  ON  COMPUTER-AIDED  DESIGN  OF  INTEGRATED  CIRCUITS  AND 
SYSTEMS,  VOL.  27,  NO.  7,  JULY  2008] 


Soundness  means  that,  if  the  tool  reports  a  property  or  assertion  is  met, 
the  tool  can  be  trusted. 

Undecidability  means  that  the  tool  might  not  be  able  to  decide  for  every 
possible  property  or  assertion  (it  is  “incomplete”). 


Booz  Allen  Hamilton 


19 


The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Programming  constraints  to  enabie 
sound  static  anaiysis 


►  Specialized  programing  or  modeling  languages 

•  Esterel/Lustre 

•  Signal 

•  Promela  (for  formal  analysis  by  SPIN) 

►  Language  subsets 

•  Escher  C  Verifier  (verifies  programs  written  in  an  annotated  C 
subset) 

•  KeY  (verifies  properties  of  programs  written  in  a  Java  subset) 

•  VeriFast  (verifies  programs  written  in  Java  or  C  subsets) 


Booz  Allen  Hamilton 
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The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Safety-Critical  Decision  Points 

►  Safety-critical  software  has  command  authority  over  potentialiy 
dangerous  system  actions. 

►  The  software  is  therefore  responsible  for  making  the  decision  to  take 
that  action. 

►  if  the  data  used  to  make  the  decision  is  corrupted  or  stale,  the 
software  can  make  the  wrong  decision  with  catastrophic  results. 

►  Design  and  code  anaiysis  of  the  software  should  be  focused  on  the 
integrity  of  the  data  used  at  each  Safety-Criticai  Decision  Point  in  the 
software. 


Booz  Allen  Hamilton 
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The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Programming  slicing 


In  computer  programming,  program  slicing  is  the  computation  of  the 
set  of  programs  statements,  the  program  siice,  that  may  affect  the 
values  at  some  point  of  interest,  referred  to  as  a  siicing  criterion. 
Program  siicing  can  be  used  in  debugging  to  iocate  source  of  errors 
more  easiiy.  Other  appiications  of  slicing  inciude  software 
maintenance,  optimization,  program  analysis,  and  information  flow 
control. 


[ l/l/zTc/ped/a  article  on  “Program  Slicing,”  March  17,  2015] 


Booz  Allen  Hamilton 
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The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Some  Targeted  Static  Analysis  Tools 


►  Proving  targeted  assertions  (model  checking) 

•  Bell  Lab’s  SPIN 

•  Carnegie  Mellon’s  NuSMV 

•  Kestrel’s  CodeHawk  (abstract  Interpretation) 

•  MathWork’s  Polyspace  Code  Prover  (abstract  interpretation) 

•  Microsoft-Inria  TLA+  Proof  System  (TLAPS) 

►  Program  slicing  tools 

•  VALSOFT/Joana 

•  GrammaTech’s  CodeSurfer 


Booz  Allen  Hamilton 
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The  Givens 

Safety  vs  Security 

Generai  Anaiysis 

Targeted  Anaiysis 

Coordination 

Conciusion 

Opportunities  for  software 
security/safety  coiiaboration 


;A]II  systems  should  be  developed  as  safe  secure  systems. . .  to  allow 
"or  a  complementary  software  skill  set  in  software  development  (tools 
and  language  dependent).  This  would  require  a  common  development 
process  rather  than  a  skill  change. . .  [R]isk  and  hazard  analysis,  for 
both  a  security  and  safety  assessment,  should  be  conducted  and 
therefore  requires  skills  from  both  arenas  . . .  Independence  of  this  skill 
. . .  may  be  required  though  to  ensure  there  is  no  bias  towards 
contradicting  risks. 

[“Safety-Critical  Versus  Security-Critical  Software.”  Dr.  Adele-Louise  Carter, 

Version  1 .0,  August  201 0,  bcs.org.uk] 
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Backup  Slides 
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Tools  to  Support  Software  Safety  Analysis 


Use  tools  to  help  analyze  the  Safety-Significant  Software  in  the 
context  of  the  Architecture,  Design,  or  Code  (leverage  those  in  use 
by  the  software  developers  or  obtain): 

•  Software  architecture  and  design  modeling  and  analysis  tools, 
such  as  those  supporting  Architecture  Analysis  and  Design 
Language  (AADL),  Unified  Model  Language  (UML),  or  Systems 
Modeling  Language  (SysML) 

•  Static  code  analysis  tools  that  support  focused  design  and  code 
analyses,  such  as  thread  race/deadlock  detection  or  program 
slicing 

•  Source  code  cross  reference  tools  that  support  searching, 
cross-referencing,  and  navigating  (forward  and  backward) 
source  code  trees 
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Background  and  Motivation 


■j 


•  Performing  multidisciplinary  design  optimization  of  a  military 
ground  vehicle  is  extremely  challenging 


•  One  challenge  is  related  to  analyzing  large,  highly  dimensional 
vehicle  design  datasets 


•  Analysis  questions  to  answer  regarding  these  datasets: 

—  Do  my  highest-ranked  designs  reside  in  multiple  regions  of  the  trade  space? 
—  How  many  promising  regions  are  there? 

—  Does  each  region  represent  variations  on  a  single  design  concept  or  multiple 
design  concepts? 

—  How  can  I  best  characterize  the  unique  features  of  each  design  concept? 
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Clustering 


•  Simply  put,  clustering  is  the  process  of  assigning  data  points  to 
groups  based  on  how  closely  their  values  are  to  a  common 
group  centroid 

•  A  way  to  group  data  that  is  highly  dimensional 

•  Different  algorithms  available 

•  Machine  learning  technique 


Simple  2-D  Clustering  Example 
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Clustering  for  Trade  Space  Design  Populations 


•  Reduce  large,  highly  dimensional  datasets  to  more  manageable, 
digestible  sizes.  This  can  make  it  easier  to  draw  conclusions 


•  Automated  way  of  quantifying  and  qualifying  design  differences  - 
characterizing;  may  help  answer  the  question  of :  "How  different 
are  the  top  ranked  vehicle  designs?" 


•  Clusters  could  be  used  to  provide  promising  vehicle  design  groups, 
and  therefore  promising  characteristics,  to  be  taken  to  the  next 
stage  of  vehicle  development 
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ERS  LRV  Trade  Space  Exploration  Project 


Objectives 

•  Learn,  evaluate,  and  provide  feedback  to  developers  of  CREATE- 
GV  and  ERS  Tools 

•  Apply  these  tools  to  the  LRV  notional  concept  vehicle  to  perform 
trade  space  exploration 

•  Develop  new  trade  space  exploration  methods  for  ground  vehicles 


CREATE-GV:  Computational  Research 
and  Engineering  Acquisition  Tools  and 
Environments  -  Ground  Vehicles 


ERS:  Engineered  Resilient  Systems 


LRV:  Light  Reconnaissance  Vehicle 


UNCLASSIFIED:  Distribution  A.  Approved  for  public  release;  distribution  is  unlimited  enbineered REanEwi stsTBOs 


LRV- A  Notional  Concept  for  a  New-Start  Vehicle 

Notional  concept  was  initially  developed  based  on  these  requirements: 

•  Crew  of  6 

•  Power  for  96-hour  mission 


•  Silent  watch,  silent  move 

•  Advanced  reconnaissance  &  surveillance  equipment  package 

•  CH-47  internal  transport  and  sling-load  transport 
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Reviewed  initial  concept 
&  requirements 


Performed  analysis  to 
build  trade  space 


Iterative 

Concept- 

[Analysis 

iPop 


Revisited  concept  & 
varied  requirements 


Performed  analysis  to 
expand  trade  space 


Generated  new  design 
set  =  new  concept 


Trade  Space  Exploration  Process 


Trade  Space  Exploration  Process 
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Designs 


I 


Trade  Space  Construction  in  ERS  TradeBuiider 


CREATE-GV  ■ 

1 

On-Road  Speed 
Off-Road  Speed 

Max  Sandy  Grade 
Off-Road  No-Go  % 
Soft-soil  mobility  ^ 

2 

3 

4 

5 

ERS  TradeBuiider 

1 

Surveillance 

2 

Crew 

3 

Stability 

4 

Silhouette 

5 

Power  density 

6 

Survivability 

7 

Transportability 

8 

Lethality 

Performance  Metrics  Performance  Metrics 

Design  Variabies  imported  from  resuit  flies  evaiuated  in  ERS  TradeBuiider 
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Trade  Space  Vehicle  Design  Characterization 


•  Common  features  from  highest-ranked  designs: 


CH-47  sling-load  transportability 
V-belly,  floating  floor,  higher  standoff 
Interior  space  for  crew  of  6  +  canine 
Full  reconnaissance  &  surveillance  kit 


Floating  Floor 
&  V-Belly  Armor 


Canine  Area 


These  are  general  features  typically  seen  in  the  top  25  ranked 
vehicles,  though  not  all  of  the  top  25  designs  had  the  same  features 
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Trade  Space  Vehicle  Design  Characterization 


•  Two  areas  where  differences  are  seen  in  the  top  25  designs  : 
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Top  25  Weapon  Option 


12 


M2-50CAL 


No-Weapon 


•  Characterizing  the  top  ranking  designs  as  a  whole  may  not  lead 
to  as  useful  conclusions  regarding  which  features  a  single 
vehicle  design  should  have 

*  We  could  be  unintentionally  characterizing  multiple  vehicle 
designs,  multiple  variants,  a  potential  outcome  when 
performing  multi-objective  design  optimization 
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Trade  Space  Vehicle  Design  Characterization 


•  Early  in  the  concept  development  phase,  the  trade  space  is 
large,  with  a  design  space  that  could  be  spanning  regions 
consisting  of  two  or  more  completely  different  vehicle  designs 

•  ...  and  this  is  not  apparent 


•  We  want  to  understand  if  potential  regions  exist  early  on  in  the 
analysis  process  to  understand  what  unique  concepts  we  may 


have 


Potential  example 
outcome  of  early  concept, 
highly  dimensional,  multi¬ 
object  design  optimization 
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Clustering  Analysis  -  Setup 


•  K-means  clustering  algorithm  used  within  R  ("Hartigan-Wong" 
version) 

•  Chose  to  generate  10  clusters  based  on  the  "within  sum  of 
squares  (WSS)"  count  selection  method 

•  Design  variables  and  characteristics  chosen  for  features: 


Weapon 

System 


Surveilla 

System 


13  features 
specified 


Suspension  characteristics 

(damping  ratio  and  ride 

frequency)  for  the  front  and 

rear  axles 

Canine 

Crew  size 

Armor  weight 


Tire  Type 
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Designs  . —  .. -  Designs 


Clustering  Analysis  -  Setup 


Design  Variables 


976  designs 


CREATE-GV 
Performance  Metrics 


ERS  TradeBuilder 
Performance  Metrics 
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Clustering  Analysis  -  Selecting  Cluster  Count 


•  Within  Sum  of  Squares  cluster  count  selection  method 
(WSS) 


CO 

CO 


...  at  976  clusters, 
the  sum  of 
squares  value 
would  equal  0 


10  Clusters 
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Performance  Score 


Clustering  Analysis  -  Results 


_  Clustered  Performance  and  Cost  Distribution 
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•  Cluster  1 

•  Cluster  2 

•  Cluster  3 

•  Cluster  4 

•  Cluster  5 

•  Cluster  6 

•  Cluster  7 

•  Clusters 

•  Cluster  9 

•  Cluster  10 


Life  Cycle  Cost  in  Miiiions 


Pareto  Front 
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Clustering  Analysis  -  Characterization 
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Plot  shows  clustering  results  using 
10  specified  clusters  for  the  976 
vehicles  designs  investigated 


Showing  3  clusters  in  the  top  25: 

Cluster  7  (C7) :  2  designs 
Cluster  8  (C8) :  14  designs 
Cluster  10  (CIO) :  9  designs 
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Vehicle  Design  Count 


Clustering  Characterization  Comparison 


•  Looking  at  the  same  two  features  as  before... 


10 

8 

6 

4 

2 

0 


Clustered  Top  25  Canine  Option 
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Cluster  10 


Top  25  Canine  Option 


before 
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Cluster  10  Designs: 

Mostly  do  not  include  a 

weapon  or  canine 


Clustered  Top  25  Weapon  Option 
11 


15 

Top  25  Weapon  Option 

13 

'ehicle  Design  Coun 
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M2-50CAL  No-Weapon 

□  No  Weapon 
n  M2-50CAL 


Cluster  8 


Cluster  10 
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Clustering  Characterization  -  Features 


•  Various  visualizations 
used  to  distinguish  the 
differences  between  the 
top  25  designs  within 
clusters  8  and  10 
concerning  their  design 
variables  and 
characteristics 


Rear  Axle  Weight  % 


“I - r 

8  10 


Tire  Type 


Cluster  8  Cluster  10 

■  16R20  iSQS.SSRZO  ■  37X12.5R16.5  ■  335_80R20 


Cluster  #  Wheelbase  Weapon  Crew  Size  Rear  Weight  Pet  Rear  DR  Rear  Ride  Freq 

CG  Height  Tire  Type  Surveiilance  System  Crew  Canine  Front  DR  Front  Ride  Freq  Hull  and  Armor 


CG  Height  (in) 


10 
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Clustering  Characterization  -  Features 


Feature 

Cluster  8 

Cluster  10 

Weapon 

Most  designs  include  the 
M2-50  Cal 

Most  designs  don't  include 
a  weapon 

Canine 

Most  designs  include  a 
German  Shepherd 

Most  designs  don't  include 
a  canine 

CG  Height 

Low 

High 

Wheelbase  Length 

Longer 

Medium 

Weight  Distribution 

More  centered  to  rear 
heavy  designs 

More  front  heavy  designs 

Front  Axle  Ride 

Characteristics 

Stiff,  Mostly  Overdamped 

Less  Stiff,  Underdamped 

Rear  Axle  Ride  Characteristics 

Stiff,  Mostly  Overdamped 

Very  Stiff,  Mostly 
Overdamped 

These  clustered  designs  were  similar  regarding  the  Tires,  Hull  and  Armor 
Weight,  Crew  Size,  and  Surveillance  System  features 
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Clustering  Characterization  -  Performance 


Various 

visualizations  used 
to  distinguish  the 
differences  between 
the  top  25  designs 
within  clusters  8  and 
10  concerning  their 
performance 


Max  Speed  (mph) 


Cluster#  V50  Speed  Max  Sand  Slope  Transportability  Score  Lethality  Score  Survivability  Score  SSF 

No-Go  %  VCI1  Max  Speed  Siihouette  Surveillance  Score  Crew  Effectiveness  Score  HP /ton 
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Clustering  Characterization  -  Performance 


Performance 

Cluster  8 

Cluster  10 

Crew  Effectiveness 

Highest  score 

Meets  requirements 

Max  Speed 

Lower  to  moderate 

Moderate  to  high 

Max  Sand  Slope 

Medium 

Medium  to  high 

SSF 

High 

Medium  to  low 

Visibility  (Silhouette) 

Larger  profile 

Smaller  profile 

Lethality 

Higher 

Lower 

These  clustered  designs  were  similar  regarding  the  Surveillance,  No-Go  %, 
veil,  V50  Speed,  HP  /  ton.  Survivability,  and  Transportability 

performance  metrics 
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Clustered  Characterization  -  Conclusions 


•  Highlighted  two  main  clusters  in  the  top  25  ranked  vehicle 
designs  and  analyzed  their  features  and  performance 

•  Instead  of  describing  one  LRV  design,  now  describing  two  LRV 
design  variations  in  the  top  25  -  two  designs  that  have  some 
distinct  differences,  but  with  similar  overall  performance  scores 


Cluster  8 


Cluster  10 


Well-rounded  design 
concerning  all  of  the  areas  of 
performance  considered 


Two  potential 
variants 


Fast,  mobile 
design,  with 
smaller  profile 
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Conclusions 


New  trade  space  exploration  process  which 
utilized  a  clustering  technique  highlighted  two 
main  vehicle  variants  out  of  a  set  of  top 
performing  vehicle  designs 


•  Clustering  is  a  promising  trade  space  analysis  process  addition  to 
help  improve  and  further  automate  trade  space  characterization 

•  Can  help  answer  important  questions  about  a  trade  space 

•  And  lead  to  improved  optimal  design  extraction  from  trade 
spaces,  and  overall  improved  concept  design  development 

•  More  to  look  into  :  clustering  technique  tuning  and  feature 
selection 
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Tradespace  Analytics  -  Set-Based  Design 


Tradespace  -  the  set  of  processes,  program  and  system  parameters, 
attributes,  and  characteristics  required  to  satisfy  mission  prof  He 


POINT-BASED 

DESIGN 


SET-BASED 

DESIGN 


The  space  spanned  by 
completely  enumerated 
design  variables 


Advanced  analytics  allow  engineers  to 
investigate  more  thorough  design 
soiutions  sets 


ERS  NDIA  SYS  ENGINEERING  CONE  -  OCTOBER  2017 


Distribution  A 


ENBINEERED  RESILEHT  SYSTEMS  2 

■  -  -.N* 


Distribution  A 


ERS  Tradespace  Concept 
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Analysis  of 
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discover  key 
performance 
parameters 
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Currently  Applied  ERS  Advanced  Tradespace  Analytics 
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•  Early  concept  tool 
•Functional/ 
component  breakdown 
' ' '  •  Explore  tradespace 

edges 
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Performance  Assessments 
Performance  Metrics 


High-fidelity  Models 
Parameter  Sweeps: 
Design  Variations 


ANALYZE 


•  Highly  computational 

•Sifts  through  millions  of  designs 

•  Refined  set  of  specifications  for 
viabie  design  soiutions 
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Distribution  A 


Tradespace  Exploration  Processes 


known  designs 

physics-based  are  used  in 


Exclusion 


Comparison 


Tradespace 
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optimum 
solutions  are 
revealed 
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contrasted 
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Decision  Anaiysis: 

Integrated  Processes  with  Trade  Analytics 


EXPERTISE  &  MANAGEMENT 
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PROGRAM 

CONSTRAINTS 


MODELS 

&DATA 


The  space  spanned  by 
completely  enumerate^  Advanced  analytics  allow  engineers  to 
design  variables^  investigate  more  thorough  design 
solutions  sets 
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Technical  Requirements  of 
Data-Driven  Decisions  Toois 


Trace 

Requirements 
and  link 
systems  to 
output 


Visualize  trades  between 
dominant  variables  and 
requirements 


Allow  novice  and 
advanced  data  exploration 


Quickly 

find 

dominant 

variables 


Tradespace  tools  must: 

•  Have  a  traceable  history 

•  Utilize  cutting  edge  search  and  decision  analytics 
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Ontology  Models: 

Consistency  in  System  Communication 


Original  system  breakdowns  by  ontologies  or  SysML,  along 
with  requirements,  are  tied  to  the  tradespace 


•  Inserts  greater  accuracy  and 
verification  into  the  analytic 
processes 

•  Passing  the  metadata  gives  us 
insight  into  how  to  analyze 
the  data 

•  Direct  mapping  via  SysML 

WBS  MILSTD-881C  (soon 

881D)  is  an  OSD-CAPE 
requirement 
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Complexity 


Distribution  A 


Data  Metrics 


Humans 

10s 


PC  Web  HPC 

1000s  100000s  Billions 


Size  of  Tradespace 
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Tradespace  Analytics  -  Data  Analysis  and 

Visualization  Tooi 


Beta-Release  Status: 

•  Supplied  acquisition  community  a 
web-based  environment  for  storing, 
visuaiizing  and  anaiyzing  data 

•  Aiiowed  for  access  and  annotation 
by  muitipie  parties  for  any  given 
iocation; 

•  Provided  the  base  for  a  coiiaborative 
decision  support  environment. 

-  Gaps  in  previous  environments  forced 
point-based  design  methodology. 

•  Successfuiiy  supported  MBSE  and 
data  fiitering 

-  Previously  available  MBSE  were 
expensive  and  resource  heavy  -  requiring 
local  resources  and  administrative 
personnel,  required  expensive  licensing 
agreements. 


Tradespace 

Analysis 


Lifecycle 

J)ata 


lission 

■pntext 


Mission 

Data 


DATA&  \ 
RESOURCES 


Physics- 

based 

models 


Enterprise 
Web  Portal 


Engineering 

Acquisition  Cost  Analysis 

Training  T&E... 
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Tradespace  Analytics  Beta  Release 
Lessons  Learned 


The  FY1 7  Beta  Release  of  TradeAnalyzer  to  a  number  of  DoD 
Users  resulted  In  important  lessons  and  changes 

•  Use  of  ParaView  Web  -  generating  interactive  visualizations  of  iarge 
data-sets  and  annotation  capabilities 

•  Roie  Based  Access  Control  (RBAC)  needed  to  execute  R-Scripts  in  a 
secured  environment;  impiementation  in  a  compiex  coilaborative 
environment  is  chaiienging. 

•  Working  on  secure  authentication  mechanisms  that  coupie  with 
customers  locai  access  controi  policies  is  an  ongoing  and  important 
DoD  issue. 
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Tradespace  Analytics 
and  Visualizations 


Available  Plotting  Options 


A  diagram  consisting  of 
rectangles  whose  area  is 
proportional  to  the  frequency  of  a 
variable  and  whose  width  is  equal 
to  the  class  Interval 


A  graph  in  which  the  values  of 
two  variables  are  plotted  along 
two  axes,  the  pattern  of  the 
resulting  points  revealing  any 
correlation  present 


A  graph  in  which  the  values  of 
three  variables  are  plotted  along 
three  axes,  the  pattern  of  the 
resulting  points  revealing  any 
correlation  present 


A  graph  in  which  the  values  of 
three  variables  are  plotted  alor 
three  axes,  with  a  mesh  alpha 
clustering  overlay 


•  '  K 

0.6  • 

2 :  NORMAL! 2ED. CGV .Mercury . OUTPUTVCI Ip A 


X:  NORMALIZED. CrewEff. Model. 


iLIZED.Afr. Mercury.  OUT  POT_HaxSp«ed_inph 


Surface  Plot  3d 

Contour 

Heatmap 

^B 

Correlation  Matrix  | 

Surface  plots  are  diagrams  of 
three-dimensional  data.  Rather 
than  showing  the  individual  data 
points,  surface  plots  show  a 
functional  relationship  between  a 
designated  dependent  variable 
(Y),  and  two  inc 
variables  (X  and  Z). 


A  contour  plot  is  a  graphical 
technique  for  representirig  a  3- 
dimensional  surface  by  plotting 
constant  z  slices,  called 
contours,  on  a  2-dimensional 
format. 


Graphical  representation  of  data 
where  the  individual  values 
contained  in  a  matrix  are 
represented  as  colors. 


Graphical  representation  showing 
the  correlation  coefficients 
betweensets  of  variables. 


I  “ 


•  •  %••••• 

\  •  .fc* 

:  .. ..  . 


•  "eV  i  M  •• 

•  Jrt  \m»K 


•\  ••  ••  • 
IWc  *1  < 


Parallel  Coordinates 


A  common  way  of  visualizing 
high-dimensional  geometry  and 
analyzing  multivariate  data. 


Pareto  Front 


A  framework  for  partially 
evaluting  a  set  of  “actions"  with 
multi-dimensional  outputs 
assuming  a  very  weak 
“desirability"  partial  ordering 
which  only  applies  only  when  one 
processes  is  better  (or  at  least  as 
good)  for  all  the  outputs.  It  is 
useful  for  reducing  a  set  of 
candidates  prior  to  further 
analysis. 


..S.S4  ^  •  •  . . 

•  ••  •••«  •  •  •  • 


plotly  JC 

Oa 

pqthon 
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Updates  to  Architecture 


•  Web  Hosted 

•  Access  Control 

•  Collaboration 

•  Shared  Notebooks 

•  Shared  Data 

•  Versioning 

•  Analytic  Packages 

•  Scalable 

•  Portable 

•  Reproducible 

•  Distributed 

•  Spark 

•  Hadoop 


It's  here! 

Anaconda 
Enterprise  5 

7/)e  //rst  doto  science  platform  built  with  (he 
entire  organization  &  development  cycle  in  mind 


Relevant  ERSHMh  Talk 


1 0:40  -  Resilient  Tools:  Building  an  Agile  Framework  for  the  Analysis  of 
Environmental  Impacts  on  Military  Systems 

Dharhas  Pothina,  PhD  -  ERDC 
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Network  Access 


ERS  USERS 


DREN 

Internet  Interface 


ERS  SEGMENT  on  the  DREN 


HPC.MIL 


Distribution  A 


Questions 
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What  is  a  Tradespace 


*  Tradespace  is  the  space  spanned  by  completely 
enumerated  design  variables.  It  is  the  potential  solution 
space. 

*  Tradespace  can  also  be  defined  as  the  set  of  processes, 
program  and  system  parameters,  attributes,  and 
characteristics  required  to  satisfy  mission  profile. 

*  The  enumeration  of  a  large  tradespace  helps  prevent 
designers  from  starting  with  point  designs  while 
allowing  them  to  investigate  more  thorough  design 
solutions  sets. 
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System-Supported  Collaboration  Supports 
Data-Driven  Decision-Making 


ERS  Tradespace  Analytics  support  Collaborative  Processes 

Engineering 

Acquisition  *  Cost  Analysis 
Training  T&E... 


Previous 

Design 

Successes, 

Lessons- 

learned 

Needs  (...iiities) 

•  Manufacturability 

•  Affordability 

•  Reliability 

•  Sustainability 

•  Usability 

•  Testability 

•  Etc. 

HPCMP 

Resources 


S&T 

Resources, 

Research 


Rapid, 
Reconfigurabie 
Systems 

Acquisition 

Soiutions 


ERS  NDIA  SYS  ENGINEERING  CONE  -  OCTOBER  2017 


Distribution  A 


ENBINEERED  RESILEHT  SYSTEMS  1 7 

■-N* 


Physics  and  Model-Based  Aerodynamic  Design  and  Anaiysis 


This  document  does  not  contain  U.S.  export  controiied  technicai  data. 


General  Atomics  Aeronautical  Systems 


Predator  A 
Piston 

(In  Production) 


Predator  B/C 
Turboprop/fan 
(Production/Dev) 


Small/Large 
UAVs 
(In  Dev) 


nreaatof  xr 


Cray  Eagle 


Cray  Eagle  Extended 
Range  (CE-er) 


\ 


International 


U.S.  ARMY 


U.S.  ARMY 


Predator  B 


MQ-9B 


Predator  c  Avenger 


U.S.  Air  Force 


Type-Certifiable 


Turbofan 


CENERJiL  JiTOMiCS 

AERONAUTICAL 
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Product  Aerodynamic  Lifecycle 


Requirements 


Conceptual 

Design 


Prelim/Detailed 

Design 


Test 


Sustainment  / 
Growth 


Aerodynamic  design  and  analysis  relevant  to 
all  stages  of  the  product  life  cycle 

Ideally  need  a  set  of  “multi”  tools 

-  Multi-fidelity  (low  ^  high  fidelity) 

-  Multi-physics  (aero  ^  aero+) 

-  Multi-cost  (sec/min  ^  days/weeks) 

-  Multi-user/org  (aero  vs.  struct  SME) 

-  Multi-product  (Aircraft  A  vs.  Aircraft  B) 


CENERJiL  JiTOMiCS 

JiERONHUTICJtL 
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Aerodynamic  Pre-Flight  Tool  belt 


Physics 

Vortex  Lattice  /  Panel 

CFD 

Wind  Tunnel  I 

Inputs 

Conceptualize  ^  Run 

CAD  ^  Mesh  ^  Run  ^  Post 

Plan  ^  CAD/Build  ^  Test  ^ 
Post 

Outputs 

Steady/Unsteady 

Linear  aero 

Quick  prelim  results 

Steady/Unsteady 

Non-linear  aero 
Validation  required 

Typically  steady  aero 

Non-linear  aero 
Established  data  source 

Scale 

(Reynolds  #) 

Full-scale 

(Inviscid  i.e.  Re^°°) 

Full-Scale 

(Flight  Re) 

Sub-scale  or  partial  model 
(Variable  Re  adds  cost) 

Compressibility 

Incompressible  or 
compressibility  corrected 

Compressible 
(Flight  Mach) 

Compressible.  Separate  tests 
depending  on  Ma 

Viscous 

Effects 

Inviscid 

or  viscous  corrected 

Typically  fully  turbulent 
Recent  RANS  transition  models 

Typically  tripped 
or  natural  transition  at  test  Re 

Geometry 

Panel  representation  and 

simple  shapes 

Geometric  complexity 
increases  meshing  cost;  smooth 

Smooth;  gaps/slots  sizes  may 
need  to  be  Re  scaled 

Propulsion 

Faired;  no  or  limited  prop 
effects 

Faired  or  flow-through; 
can  model  propulsion  effects 

Faired  or  flow-through; 
separate  tests  for  prop  effecfs 

Environment 

Modeled  in  farfield 

Modeled  in  farfield 

Corrected  for  funnel  effects 

_ 

This  document  does  not  contain  U.S.  export  controiied  technical  data. 
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Requirements  /  Conceptual  Design 


•  Semi-empirical  methods  drive 
requirements  and  sizing 

-  High  level 

-  Grounded  in  actuals 

-  Good  for  derivative  designs 

-  Good  for  high  level  trades 

•  Opportunities 

-  Multi-fidelity  framework  at  GA-ASI 

-  Others  successfully  options  exist 
e.g.  MIT  TASOPT 


Multi-fidelity 

Framework 


Common 

Parametric  Definition 


GENERAL  ATOMICS 

JiERONHUTICJtL 
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Preliminary  /  Detaiied  Design 


CFD  and  wind  tunnel  test  drive 
design 

-  Analysis  for  design  trades 

-  Test  for  database  generation 

-  Test  for  pert  verification 

Challenges 


control  surfaces  and  props 


-  Managing  multiple  models... 
CREATE-AV  enabling  multi¬ 
disciplinary  analysis 

-  Physics!.,  the  PANS  plateau 
LES/DDES  still  costly 


Flop-Tail  Interaction 
DDES  Simulation 


CBNERJiL  JiTOMtCS 

MERONMTICML 
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Prelim  /  Detailed  Design  (Cont.) 


•  Challenges  (Cont.) 

-  Scalability...  Wind  tunnel 
cheaper  than  CFD  for  large 
databases. 

-  Trust...  CFD  meshing  treated  os 
an  “art.”  Mesh  convergence  + 
Solution  accuracy.  Test 
validation  remains  essential. 

-  Expectations...  CFD  not  fast 
enough  to  be  in-exact. 

-  Process...  CFD  treated  as  virtual 
vs/ind-tunnel. 


- SST 

- SA 

- SA-RC 

SA  (other) 

EARSM  .-ST 


5  Counts 


•  GA-CFD  Group  k-w  SST, 
Vassberg  Multiblock  Grid 


----NTFTest  197  Runs 
44,51,53 


AIAA  Drag  Prediction 
Workshop  (DPW5) 

Graphic  from:  https://aiaa- 

dpw.larc.na$a.qov/Workshop5/pre$entation$/DPW5  Presentation  Files/I 4  D 

PW5%20$ummarv-Draft  VZ.odf 


5E-05  0.0001  0.00015 

GRDFAC  =  1/GRIDSIZE*''^* 


■|0.028 
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Test 


•  Pre-test  predictions  inform  test 
focus  areas 

•  Test  helps  CFD 

-  Separated  flows 

-  Interaction  effects 

-  Transition 

•  CFD  helps  test 

-  Wind  tunnel  corrections 

-  Propulsion  effects 

-  Aero-stotic  effects 


,  Sub-scale  Wind  Tunnel 

'  ,Mi|;  iJiiVV'i  ■, 


Laminar  bubbles  match 


CFD  Test 


Tunnel  Effects 


GENERAL  ATOMICS 

JiERONMTICML 
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DENSITY  ALTITUDE  -  FEET 


Closing  the  Loop  on  Performance 


FLIGHT  ENVELOPE  -  KTAS 
STANDARD  DAY 


AIRSPEED -KTAS 


All  aero  models  contribute  to: 

•  Understanding  of  aircraft  flow-field 

•  Support  modeling  for  pert  and  S&C 

•  Air-data  integration 


RANs 

Aero 


CENERJiL  JiTOMiCS 
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Sustainment  /  Growth 


•  New  tools  provide  opportunities 
to  improve  existing  systems  and 
match  evolving  customer  needs 

•  GE  ^  GE-ER  Case  Study 

-  GE  double  slotted  flop  designed 
with  2D  CFD 

-  GE-ER  reconfigured  existing 
hardware  to  a  single  slotted  flop 
with  3D  CFD 

-  Wind  tunnel  and  flight  test  in 
both  coses 

-  Meet  current  customer  needs 


Physics-based 
model  used  to  test 
flop  concepts  (3D 
+  transition 
modeling) 


Rejected 
;^^oncept  with 
separated  flaps 


^  Final  design; 
separation  only 
behind  flap 
fairings 


CENEmXL  iXTOMtCS 
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Future  Needs 


•  Medium  fidelity  needs 

-  Fast  3D  methods  (can  include  fuselages) 

-  Non-linear  unsteady  options  (damping  deriv,  loads  spectra) 

•  Promising  Candidates 

-  Coarsely  auto-meshed  RANS/URANS  with  wall  functions 

-  Auto-meshed  Euler+IBLT3 

-  Probabilistic  multi-fidelity  methods  like  Kriging 

•  High  fidelity  needs 

-  More  efficient  algorithms  (e.g.  multi-grid) 

-  Less  reliance  on  hardware  solutions  (costly) 

-  Faster  CAD  clean-up  (time  consuming) 

•  Transition  modeling  essential  for  GA-ASI 

-  RANS  based  models  promising  from  computational  cost  perspective 

-  Need  models  robust  to  Re  5e5-l  0e6  (current  y-ReG  not  there) 

-  Natural  transition  covering  TS,  CF,  laminar  bubbles,  attachment  line 
contamination 

-  Forced  transition  covering  trip,  surface  roughness/defects 

-  Non-dissipotive  methods  for  high  level  for  freestreom  turbulence  in 
RANS 
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Objective  and  Outline 


Provide  ERS  Lifecycle  Cost  (LCC)  development  plan 
and  methods  for  linking  cost  models  to 
performance  models  for  generating  largescale 
tradespaces 


•  Objective 

•  Background 

•  Cost  Estimating  Techniques 

•  Cost  Analysis  Use  Case 

•  Surrogate  Model  Creation  Method 

•  Low-Cost  Attritable  Aircraft  Use  Case 

•  ERS  Cost  Model  Development  Plan 

•  Summary 

•  Questions 
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Background 


A  goal  of  the  Engineered  Resilient  Systems  (ERS)  Program  is  to  create  a 
capability  for  linking  cost  and  performance  models  for  early  concept  exploration 

of  design  alternatives 


Affordability 
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daptability 

Rapid  Response  to 
Emerging  Threat 


Effectiveness 

Confidence  in  Decisions 


On  Time,  on  Budget 


Reduce 
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Background 


Affordability  Analysis 
(Pre-Milestone  A/B) 


•  Determine  Affordability 
Goals/Caps 

•  Estimate  Program 
Lifecycle  Cost 

•  Establish  Cost  Targets 

•  Analyze  Cost/Performance 
Trades 


Committed  Lifecycle  Cost 


in 

O  80% 
0) 

o  70% 
O 

^  fino/ 


50% 


O  40% 
(1> 

Q. 

(D  30% 
> 


E 

3 

o  10% 


Committed  Costs 


85% 


70% 


20-100X 


95% 


500  -1000X 


Prod/Test 


3-6X 


Develop 


Design 


Concept  p 


8% 


15% 


50% 


20% 


Operations 

Through 

Disposal 


100% 


Time 


A  A 


Materiel 

Solution  Technology 

Analysis  Maturation  &  Risk 


(Program 

Initiation) 


Engineering  and 
Manufacturing 
Development 


Reduction 


.  ‘.Post-  /\ 
V  PDRA  V 


Post- 

CDRA 


A 

IOC 

FOC 

Production  & 

Operations  & 

Deployment 

Support 

LRIP/IOT&E  A 

V  Rtvitw 

Reference  DoDI  5000.02  Defense  Acquisition 
Life  Cycle  Compliance  Baseline 
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Cost  Estimating  Techniques 


Analogy 


•  Quick,  inexpensive,  easy-to-change 

•  Subjective,  not  precise,  poor  comparison  between  new  and  old  systems 

•  Typically  used  pre-Milestone  A  through  Milestone  A 


Parametric 


•  Cost  estimating  relationships,  inexpensive,  easy  to  do  “what-if  drills 

•  Moderately  subjective,  precision  only  as  good  as  databases 

•  Typically  used  pre-Milestone  A  through  Milestone  B 


Engineering 


•  Very  accurate  in  later  stages  of  EMD,  limited  subjectivity,  uses  WBS 

•  Very  expensive,  very  time  consuming,  “what-ifs”  are  difficult 

•  Typically  used  Milestone  B  through  post-Milestone  C 


Actual  Costs 


DAU  ACQ101 
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•  Limited  subjectivity,  very  accurate 

•  Limited  actual  cost  data,  very  expensive,  very  time  consuming, 
“what-ifs”  are  difficult 

•  Typically  used  Milestone  C  through  post-Milestone  C 
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Cost  Analysis  Use  Cases 


Use  Cases  ERS  Partner 

Create/ Adapt 

Cost  Model 

1  -  Manual  CER:  User  manuallv  enters  Cost 

Estimating  Relationships  (CER)  to  build  a  cost  model 

UAS 

(NCCA,  UAS  Handbook) 

2  -  Existina  Menu:  User  choses  an  existina  cost 
constraint  component  and  adjusts  (calibrates)  for 
soecific  cost  Generation 

Helicopter 

(GTRI,  Commercial  rotorcraft 
cost  model) 

3  -  Historic  Cost  Data:  cost  model  from  user 
provided  historic  cost  data 

Ground  Vehicle 

(TACOM,  CADE  data) 

Link  Existing 

Cost  Model 

4  -  Existina  Model  Surroaate:  Allows  user  to  provide 
an  existing  cost  data  set  derived  from  any  source  to 
generate  meta  model  for  cost  domain  tradespace 
generation  (surrogate  cost  modeling) 

Surface  Ship 

(NSWC  Carderock,  Surface 
Combatant  Performance 
Based  Cost  Model) 

5  -  Excel  Cost  Model:  Allows  user  to  provide  an 
existing  excel  based  cost  model  to  link  to  tradespace 
generation 

Un-Manned  Aircraft 

(AFRL  -  LCAAT) 

6  -  COTS  Cost  Model:  User  provides  a  COTS 
integrated  tools  model 

[development  pending] 
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Surrogate  Model  Creation  Method 


Connecting  cost  models  to  other  tradespace  models 


^Python  Wrapper/Parseiy 


Use  existing 
spreadsheet  cost 
model 


Use  Monte-Carlo 
techniques 


Surrogate  Model  J 


Generate  surrogate- 
regression  model 
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Low-Cost  Attritable  Aircraft  Use  Case 


Surrogate  Method 


Current  Method* 

f  \ 

Computer-language 
cost  model  derived 
from  spreadsheet 
to  MATLAB  or 
Python 

V  J 


•  4  months  development 

•  Slow  response  to  changes 


□  Aeroelasticity 

□  Structural  sizing 

□  Cost 

□  Stability  &  Control 

□  Multi-Fidelity 

□  Parametric  Analysis 
Optimization 


^  Surrogate  Model  J 


1 0OX  reduction  in  cost 
model  integration  period 


•  24  hours  development 

•  Quick  response  to  changes 


*Not  typical 
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ERS  Cost  Model  Development  Plan 


FY17 


FY18 


FY19 


IQ  2Q  3Q 

4Q  IQ 

2Q  3Q  4Q 

IQ  2Q  3Q  4Q 

LCC  TradeBuilder  Module  vl.O 
(parse  constraints  from  models, 
CERs,  add  functions  via  GUi) 

LCC  TradeBuilder 
Module  vl.l  (link  LCC 
Python  and  surrogate 
models  ) 

LCC  TradeBuilder  Module  v2.0 
(link  CADE  Enterprise  for  historic 
data  and  increase  DoD  system 

LCC  model  portfolio,  cost  as  a 
function  of  llities  (RAM) 

LCC  TradeBuilder  Module  v3.0  (Use  CADE 
Enterprise  data  to  build  LCC  models  by 
analogy,  additional  total  cost  influence  from 
llities  (Mfg/Production,  etc.) 

UAS (NCCA) 


Manual  CER 
Completed 


Surrogate  cost  modeling  R&D 


Transform  Spreadsheets 


CADE  Enterprise 


I  Historic  data  &  CERs 

/  V 


LCC  by  Analogy 


Historic  data  &  CERs 


Light-Wheeled 
GV  model 
(TACOM) 


,  Initial  rnmmnHitv  Arpac  \ 


LCAAT  (AFRL) 


Python  Models  built 
from  Spreadsheets 


Universal  Wheeled  GV 
model  (TACOM) 


Generalizable  component 
built  directly  in  python 
framework 


Completed 


Initial  Commodity  Areas  \ 
(MILSTD-881) 


GV  model  (TACOM) 


Aircraft-UAS  model 
(AFRL&  NAVAIR) 


Surface  Ship  (NSWC) 


Models  (Python, 
Spreadsheets,  etc.] 

rr 


Remaining 
Commodity  Areas 
(MILSTD-881  Systems) 
(priority  TBD  of  13) 


Models  (Python, 
Spreadsheets,  etc.) 


□ 


Create/Adapt 


Cost  Model 


Link  existing 
*■"'  Cost  Model 


Reliability,  Availability,  Maintainability  R&D  (w/Boeing,  AFRL,  AMSAA,  MSU) 


Mfg/Producibility,  Affordability  (MSU,  industry,  OSD-CAPE) 
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Summary 


•  DoDI  5000.02  identifies  the  requirement  at 
Milestone  (MS)  A  for  an  Affordability  analysis  in 
addition  to  a  cost  analysis  and  is  driving  more 
accurate  cost  analysis  to  the  left 

•  ERS  is  developing  methods  to  better  integrate 
cost  models  into  conceptual  tradespace 
exploration  using  existing  models  or  surrogate 
models 


Ground  Vehicle 


•  Surrogate  modeling  methods  show  promise  to 
greatly  accelerate  the  integration  process  into 
tradespace  exploration  for  pre-MS  A  &  at  MS  A 

•  The  ERS  cost  model  development  plan  strives  to 
provide  a  capability  for  all  system  commodities 
supporting  all  Services  and  OSD-CAPE 
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An  Adaptive  Automation  Approach  for 
UAV  Ul  Concept  Development 
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Human  Systems  Engineering  Branch 
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Problem.  Solved. 


High  loss  rate  of  U.S.  Military  UAVs 

Numerous  ergonomic  /  automation  causal  factors 
(Source:  USAF  SAB): 

•  80%  of  Predator  mishaps  involved  human  error  due 
to  fundamental  design  issues. 


•  Warning/status  messages  buried  layers  deep. 

•  Complex  automation  (22  steps  to  turn  on  the 
autopilot  on  the  Predator). 

•  $4.5M  Predator  lost  due  to  pilot  accidentally  selected 
the  engine  kill  switch  instead  of  the  landing  gear 
switch. 

Analogous  in  terms  of  maturity  to  early  manned 
cockpit  design  (systematic  control  shape  coding 
analyses  fixed  a  spate  of  B-17/B-25  crashes). 

Need  a  Systems  Engineering  approach  to  higher 
order  human/automation  system  design. 


m 


g  Emergent  Requirements 
Ihe  Need  for  Automation 
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New  UAV  Combat  Missions: 

•  Airborne  Electronic  Attack  (AEA) 

•  Air  to  Ground  (A/G) 

•  Air  to  Air  (A/A) 

New  User  Interface  Goals: 

•  Single  Pilot  for  multiple  UAVs  _ 

•  Multiple  user  interactions  (ground  troops,  manned  air) 


UAV  kill  by  Tactical  High 
Energy  Laser  (THEL)  2011 


Derived  Requirements  Mandate  the  use  of  Automation: 

•  Single  pilot  mismatch  with  available  attention  span  over  multiple  vehicles  and  multiple 
users. 

•  Human  reaction  time  mismatch  (reactive  jamming  of  enemy  radar  pushes  automated 
response  requirements) 

•  Human  computational  limit  reached  (pilot  is  overmatched  trying  to  compute  fuel  burn  vs. 
rerouting  requirements  for  signature  management,  etc.). 


UAVliiJi'MJ 


|Automated  Capability 


Georgia 
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UAV:  “an  aircraft  or  balloon  that  does  not  carry  a  human  operator  and  is 
capable  of  flight  under  remote  control  or  autonomous  programming.” 

(US  DoD  Definition:  JP  1-02) 


•  Current  UAVs  have  very 
limited  autonomy  (e.g. 
preprogrammed  flight  to 
regain  a  lost  link,  auto  land). 


•  Designers  are  struggling  with 
adding  more,  incrementally. 


[nate  -  and  what  to  NOT. 
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•  The  appropriate  Systems  Engineering  question  is  not  “how  to  design  man  out”, 
but  rather  “which  functions  and  tasks  are  appropriate  to  automate,  and  how?”. 

•  Factors  include: 

•  Tactically  significant  timelines 

•  Latency  in  the  control  loop  (Observe/Orient/Decide/Act  -  OODA) 

•  Need  for  human  oversight  and  control  -  with  weapons  releases. 


•  The  next  step  is  to  recognize  the  need  for  automation  to  manage  automation 
itself. 


Operator  Role  Theory  of  Automation 

(Folds,  1995) 


Georgia 
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NO  EQUIPMENT 
IN  THE  LOOP 


INCREASING  USE  OF 
AU  TO  MATION 


,NO  OPERATOR 
IN  THE  LOOP 


ir 


“DIRECT 

“MANUAL 

PERFORMER” 

CONTROLLER” 

REGION 

REGION 

>  HUMAN  CLOSES  LOOP 

‘  HUMAN  CLOSES  LOOP 

.  CONTROL  LOOP 
COMPONENTS 

PREOOM  INANTIY 

HUMAN 

.  CONTROL  LOOP 
COMPONENTS  AREA 
MIXTURE  OF  HUMAN 

AND  MACHINE 

“SUPERVISORY 

CONTROLLER” 

REGION 

“EXECUTIVE 

CONTROLLER” 

REGION 

‘  HUMAN  OR  MACHINE 

•  MACHINE  CLOSES 

CLOSES  LOOP 

LOOP 

*  CONTROL  LOOP 

•  CONTROL  LOOP 

COMPONENTS  ARE 

COMPONENTS  ARE 

P  REDOMINANTLV 

MACHINE 

1 

MACHINE  ONLY 

.  HUMAN  MAY  START  OR 

STOP  FUNCTION 

J 


stems  Approach 


•  Need  a  system  of  systems 
engineering  approach  across 
appiications  -  to  adaptive  automation. 

•  Perform  MTA/Task  Decomposition 
and  appiy  Operator  Roie  Theory  to 
determine  mission  eiements. 

•  Determine  which  eiements  wili 
exceed  human  spans  of  capabiiity. 

•  Determine  the  modes  of  interaction 
between  automation,  and  the 
overarching  control  ioop  tasks. 

•  Determine  where  Executive  level 
automation  is  best  suited  to  arbitrate 
or  interpoiate  or  monitor,  and  where 
the  tasks  are  best  suited  for  humans. 
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X-47B  UCAS_D 

Carrier  Control  Integration 


nt  Example 
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The  Executive  Agent 

•  Monitors  automation  managers  within  UAVs. 

•  Monitors  coordinated  tactics  across  UAV  piatforms. 

•  Compares  weighted  impacts  of  confiicting  automation. 

•  Auto  performs  defined  tasks  /  aierts  pilot  for  other  tasks. 


J 

+  N 

1 

1 

The  Datalink  Manager 

The  Signature  Manager 

•  Monitors  dataiink  iatency  and 
quaiity  against  calcuiated  range. 

•  Monitors  ownship  muitispectrai 
vis  against  known  threat  sensors. 

•  Multipie  iinks  (UAV/UAV, 
UAV/manned,  UAV/GCS,  etc.) 

•  Continuousiy  computed  during 
maneuvering. 

•  Alerts  when  nearing  iost  link. 

•  Aierts  when  near  high  Pd. 

•  Sets  fiight  path  to  regain  iink. 

•  Sets  fiight  path  to  avoid. 

the  OODA  Loop 
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•  Monitor  (“Observe/Orient”) 


•  Adjudicate  (“Decide”). 

•  Recommend  (or  “Act”). 

•  Inform:  elevate  urgent 
advisories  (would  inform,  then 
prompt,  then  warn). 

•  Perform  specific-to-general 
reasoning  related  to  induction, 
synthesis,  and  integration  tasks. 

•  Perform  general-to-specific 
reasoning  related  to  deduction, 
analysis,  and  differentiation. 


•  Return  the  pilot  to  the  role  of  a 
tactician. 


Summary 
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•  The  piecemeal  use  of  automation  may  be  worse  than 
having  none. 

•  By  equipping  proposed  future  multiple  combat  UAV 
control  systems  with  agile,  Executive  level  controllers 
which  can  rapidly  perform  multivariate,  weighted 
arbitrations  between  systematically  integrated 
automation,  time  critical  combat  tasks  can  be  met  within 
the  multiple  UAV  control  paradigm. 


An  Adaptive  Automation  Approach  for  UAV  UI  Concept 

Development 


Jeff  O’Hara  Stuart  Michelson 

404-407-8507  404-407-6162 

Jeffrey.  ohara@  gtri.  gatech.edu  stuart.michelson®  gtri.gatech.edu 


Georgia  Tech  Research  Institute 
Electronic  Systems  Laboratory 
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Atlanta,  GA  30332-0840 


Abstract 

Despite  decades  of  industry  experience  in  the  design  of  Unmanned  Aerial  Vehicle  (UAV)  control 
systems  and  their  user  interfaces,  a  combination  of  factors  persist  that  produce  a  significant  and 
unacceptable  loss  rate  of  UAVs  due  to  poor  user  interfaces.  One  significant  element  is  the  current 
focus  of  human  systems  design  on  lower-order  User  Interfaces  (UI)  at  the  expense  of  investing  in 
the  design  of  an  adaptive  higher  level  integration  to  relieve  inattentive  or  overtaxed  operators  of 
significant  functionality  as  required,  and  to  perform  time-critical  tactical  tasks  which  humans 
cannot  perform  or  for  which  they  are  not  well  suited.  The  approach  proposed  is  one  which  defines 
the  respective  roles  of  user  interactions  with  adaptive  policy  manager  automation  to  address  the 
loss  of  vehicles  and  mission  failures.  Specific  policy  manager  automation  elements  are  explored 
which  will  enable  the  system  to  flexibly  assume  or  release  UAV  vehicle  or  systems  functionality 
based  on  operator  action/saturation  in  a  number  of  mission  areas.  A  notional  Executive  automation 
controller  design  approach  is  outlined  to  meet  time  critical  information  integration  and  mission 
task  requirements. 


Introduction  and  Historical  Background 

Despite  decades  of  industry  experience  in  the  design  of  Unmanned  Aerial  Vehicle  (UAV)  control 
systems  and  their  user  interfaces,  a  combination  of  factors  persist  that  produce  a  significant  and 
unacceptable  loss  rate  of  UAVs  due  to  poor  user  interfaces.  By  way  of  comparison  to  the 
progression  of  manned  aircraft  pilot  vehicle  interfaces,  the  UAV  UI  field  has  failed  to  progress  as 
rapidly,  being  somewhat  stalled  at  an  equivalent  of  a  1940’s  state  of  the  art  with  design  foci  on 
improved  detailed  level  UI  (menus,  knobs,  switches,  screens),  rather  than  on  addressing  systematic 
higher  order  user-system  automation  design. 

In  the  1940s,  manned  aircraft  human  engineering  underwent  a  radical  change  in  design  philosophy 
with  the  work  of  human  factors  engineering  pioneers  such  as  Alphonse  Chapanis,  who  applied 
engineering  psychology  to  correct  basic  cockpit  design  flaws.  The  classic  example  of  application 
of  early  engineering  psychology  analyses  is  the  effort  to  mitigate  a  rash  of  bomber  gear  up  crash 
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landings.  Human  factors  engineers  redesigned  landing  gear  handles  to  be  shaped  like  wheels  and 
reshaped  flap  handles  shaped  like  flap  handles  for  tactile  discriminability  by  pilots  who  were 
visually  focused  on  performing  landing  tasks.  These  were  point  design  solutions,  but  were 
systematically  applied  through  the  cockpit  and  were  eventually  incorporated  into  the  military 
standard  system  (Roscoe,  1995). 

A  systematic  review  in  201 1  by  the  U.S.  Air  Force  Scientific  Advisory  Board  found  a  number  of 
significant  ergonomics  and  automation  deficiencies  in  several  current  UAV  Ground  Control 
Systems  (GCS),  including  poorly  mechanized  autopilot  interfaces  as  well  as  “classic”  pilot  vehicle 
interface  deficiencies.  One  example  recalled  the  1945  bomber  crashes;  the  crash  of  one  $4.5 
million  Predator  UAV  was  directly  caused  by  a  pilot  mistakenly  choosing  the  “kill  engine”  switch 
instead  of  the  adjacent  landing  gear  switch  (Morely,  2012).  That  a  Predator  pilot  was  even  able  to 
mistake  (let  alone  be  allowed  to  actuate  in  flight)  the  "kill  engine"  switch  for  the  landing  gear 
switch  would  seem  to  indicate  the  lack  of  a  systems  engineering  analytical  approach  to  user 
interface  requirement  definition. 

Other  studies  have  confirmed  the  apparent  lack  of  a  systematic  design  approach.  A  2007  Air  Force 
Research  Lab  study  found  that  up  to  80%  of  Predator  mishaps  alone  involved  human  error, 
including  poor  documentation,  crew  coordination  mistakes  and  training,  and  serious  fundamental 
human  factors  design  issues  with  GCSs.  For  example,  it  apparently  took  22  key  strokes  to  turn  on 
the  autopilot  on  early  Predators;  warning,  caution  and  advisory  messages  were  buried  under  layers 
of  noncritical  interfaces,  resulting  in  situations  where  the  pilot  receives  few  if  any  alerting  cues  to 
emergencies.  More  than  400  US  UAVs  have  crashed  since  2001  (including  midair  collisions)  and 
due  to  these  causes,  which  contributed  to  lack  of  pilot  awareness  of  or  correct  responses  to  weather, 
fuel  status,  data  link  strength,  and  high  terrain  (Craig,  2012). 

Looking  forward,  UAV  missions  are  expanding  and  multiplying  into  roles  (such  as  Airborne 
Electronic  Attack  and  Air  to  Air  engagements)  which  stress  rapidity  of  decision  making  in  a 
complex  shifting  combat  environment.  Emergent  warfighter  UAV  design  goals  are  trending 
toward  requirements  for  single  user  command  and  control  of  multiple  heterogeneous  UAV 
platforms  with  separate  mission  taskings,  as  well  as  requirements  for  cooperative  control  between 
a  GCS  and  an  off  board  user  (such  as  a  front  line  soldier  or  pilot).  A  Human  Systems  Integration 
(HSI)  design  approach  limited  to  lower  order  point  design  switch  and  display  issues  or  merely 
complying  with  military  standard  compliance  audits  does  not  address  the  systems  engineering 
challenges  from  these  needs.  These  new  requirements  present  more  challenging  problems  such  as 
issues  with  single  user  task  saturation  and  vigilance  and  how  user  system  automation  can  augment 
a  human  user  to  prevent  mishaps  and  enable  mission  success.  This  paper  will  summarize  an 
approach  to  provide  a  framework  for  an  adaptive,  operator  centric  automation  framework  for 
future  and  retrofit  naval  UAV  designs. 

The  approach  recommended  is  two  faceted;  the  first  is  the  need  for  individual,  adaptive  automated 
policy  managers  focused  on  specific  mission  tasks  (especially  those  needing  rapid  calculation  or 
constant  monitoring).  The  second  is  the  need  for  an  overarching  Executive  manager  to  provide 
rapid  arbitration  and  coordination  during  time-critical  combat  operations.  The  end  goal  is  to  return 
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the  user  to  the  role  of  tactician,  automating  first  order  calculations  (e.g.  fuel,  terrain  avoidance)  but 
with  a  higher  order  automated  process  to  ensure  a  coordinated  response  to  human  tactical  direction. 

Progress  towards  Adaptive  GCS  Automation 

Two  historically  prevalent  approaches  to  UAV  GCS  design  have  been  followed.  One  approach 
focused  on  provision  of  controls  duplicating  manned  aircraft  interfaces  (e.g.  the  approach  used 
from  1940’s  designs  up  through  the  MQ-1  Predator).  The  other  provided  direction  of  the  vehicle 
through  graphical  map  cues  (evolving  from  hard  copy  strip  charts  to  present  day  point  and  click 
graphical  interfaces  to  direct  flight  to  a  point).  Either  approach  offers  the  potential  for  the 
uncoordinated  application  of  multiple  instances  of  automation  (e.g.,  an  automated  route  planner 
will  disagree  with  an  automated  terrain  avoidance  system  -  and  will  present  disharmonious  results 
to  the  user  from  separate  displays).  The  risk,  then,  is  that  attempts  to  add  automation  to  GCS 
designs  (within  either  design  paradigm)  will  impose  additional  new  tasks  and  roles  on  the  user  to 
monitor  multiple  automated  systems  across  multiple  vehicles,  thus  increasing  the  risk  of  significant 
error.  For  example,  trending  UAS  human  errors  have  been  noted  to  include  (Johnson,  2007): 

1.  Loss  of  operator  situational  awareness  (SA)  of  airspace  and  traffic. 

2.  Operator-induced  Air  Vehicle  loss  of  fuel/loss  of  link,  leading  to  vehicle  loss. 

3.  Loss  of  operator  SA  of  altitude,  airspeed,  vehicle  status,  and  clearance  to  terrain. 

Operator  Role  Theory  (Folds,  1995)  posits  a  spectrum  of  human  and  automation  shared  roles  in 
systems  control  (see  Figure  1,  below).  Where  no  automation  is  present,  the  user  is  acting  in  a 
“Direct  Performer”  situation.  With  automation  present  but  with  the  user  performing  information 
synthesis  and  control  of  the  system,  the  system  is  running  in  a  “Manual  Control”  region.  With 
predominantly  automated  control  loop  processes  and  user  monitoring  and  adjustment,  the  system 
is  in  a  “Supervisory  Controller”  region,  and  finally,  in  the  “Executive  Controller”  region  of 
automation,  the  human  is  not  in  the  control  loop  at  all,  save  for  a  start/stop  function 
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Figure  1  Continuum  of  Operator  and  Automation  Roles 
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The  classic  example  of  Executive  level  control  is  cell  phone  tower  switching,  which  takes  place  at 
an  Executive  level  (human  interaction  with  this  automated  element  is  generally  limited  to  seeing 
the  signal  strength  bar  on  their  phone).  Currently,  GCS  designs  incorporate  a  mix  of  automation 
from  in  various  automation  control  regions,  with  varying  success.  The  move  towards  multiple 
UAV  GCS  control  will  only  exacerbate  existing  problems  without  adoption  of  a  new  element  of 
automation  to  aid  the  user  in  automation  management._Newer  GCS  designs  are  undertaking  to 
provide  adaptive  automaton  which  provides  tools  for  automatic  flight  routing,  route  deconfliction, 
and  calculation  of  weapons  engagement  zones,  SAM  shot  avoidance  cues,  and  so  forth  based  on 
integrated  “at  a  glance”  presentations  (Johnson,  2007). 

Mission  Growth  Forces  an  Approach  with  an  Executive 

As  with  the  cell  phone  example,  the  Executive  automation  role  is  well  proven  in  manned  combat 
aircraft.  Airborne  electronic  warfare  jammers  react  immediately,  for  example,  to  defeat  incoming 
enemy  missiles  by  automatically  applying  radar  jamming  techniques.  The  system  executes  the 
protective  action  because  the  pilot  doesn’t  have  the  reaction  time  (let  alone  the  surplus  workload 
capacity)  to  manually  employ  the  equipment.  Particularly  for  pilots  who  may  be  tired  or 
inattentive,  the  sudden  leap  in  activation  from  being  a  system  monitor  to  dealing  with  an 
emergency  can  lead  to  lapses  and  errors.  Thus,  a  higher  level  requirement  exists  for  a  controller 
capability  which  looks  across  automated  subsystems  for  multiple  UAVs,  accessing  data  to 
predictively  analyze  trends  and  threats  in  a  coordinated  manner,  without  the  potential  for  boredom 
or  fatigue. 

To  match  the  required  UAV  UI  demands,  a  comprehensive  shift  to  a  system  of  systems  engineering 
approach  to  adaptive  automation  -  across  applications  -  is  recommended.  With  multiple  UAVs 
aloft  in  a  highly  dynamic  battlespace  (where  UAVs  may  be  used  not  just  for  long 
counterinsurgency  patrols,  but  as  targeting  and/or  weapons  platforms  in  air  to  air  combat), 
automation  needs  to  be  considered  as  more  than  a  family  of  decision  making  tools,  but  as  an 
integrated  system  itself.  A  human  systems  engineering  approach  which  applies  operator  role  theory 
(Folds,  1995)  to  define  a  UAV  system  of  systems  will  effect  an  order  of  magnitude  improvement 
in  combat  efficiency  and  effectiveness._The  approach  proposed  specifically  advances  the  definition 
of  multi-mission  adaptive  automation  to  address  the  impacts  of  (1)  highly  complex  mission  tasking 
(2)  too  many  vehicles  to  manually  monitor  at  once  and  (3)  short  engagement  timelines. 


Elements  of  the  Integrated  Solution:  Policy  Managers  and  an  Executive 

Automation  should  relieve  humans  from  boring  housekeeping  tasks,  prevent  their  inattention  or 
raw  information  saturation  from  causing  loss  of  vehicle  and  mission  failure  conditions,  and  allow 
humans  to  do  that  which  they  do  best  (make  tactical  judgments).  Specific  automation  “policy” 
managers  should  be  considered  for  collaborative  integration  in  a  fused  GCS  implementation.  Many 
automation  elements  have  already  been  fielded  as  separate  tools  in  manned  and  unmanned  aircraft. 
However,  to  implement  enough  of  them,  over  multiple  UAVs,  with  newly  emergent  requirements 
for  tactical  engagement  accuracies  and  timelines,  additional  Executive  level  automation  is  needed. 

Each  policy  manager  has  a  role  to  play  as  individual  automated  elements  under  an  Executive, 
which  would  supplement  the  monitoring  and  arbitration  task  set  currently  allocated  to  the  human. 
An  Executive  would  be  able  to  quantitatively  perform  that  role  across  multiple  UAVs,  and  would 
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be  able  to  meet  far  tighter  accuracy  and  speed  requirements.  The  Executive  must  be  able  to  resolve 
a  best  fit  solution  for  the  active  UAV  platforms  given  preplanned  mission  constraints  by 
performing  multivariate,  weighted,  arbitrations  across  the  lines  of  the  subordinate  policy 
managers.  Example  potential  individual  automation  elements  include  Auto  Ground  Collision 
Avoidance  System  (AGCAS)  Protection,  Auto  Traffic  Collision  Avoidance  Protection,  Auto 
Envelope  Protection,  Auto  Airspace  Protection,  Auto  Datalink  Protection  and  Auto  Signature 
Protection  (among  a  host  of  other  functions).  It  is  useful  to  examine  how  two  (a  Datalink  Manager 
and  a  Signature  Manager)  interact. 

The  Datalink  Manager  monitors  established  UAV  to  GCS,  UAV  to  UAV,  and  UAV  to  manned 
mission  partner  datalink  latency  and  strength  against  calculated  range  limits.  It  then  provides  a  real 
time  calculated  assessment  of  the  probability  of  loss  of  link(s)  as  well  as  quality  factors.  (Link 
latency,  as  an  example  quality  factor,  will  impact  the  ability  of  the  vehicle  to  perform  time  critical 
tactical  tasks).  Based  on  this,  as  well  as  the  availability  of  alternative  links,  this  policy  manager 
automatically  shifts  and  configures  data  links  In  an  integrated  automation  system,  the  Datalink 
policy  manager  will  need  arbitration  with  the  Signature  and  other  managers  to  regain  signal  while 
ensuring  the  “lost”  AV  avoids  maneuvers  which  compromise  detection  or  survivability. 

The  Auto  Signature  Protection  manager  provides  real  time  computed  signature  management  to 
ensure  that  the  UAV  remains  either  undetected  or  unengageable  by  threat  systems.  Based  on 
preplanned  settings,  the  Signature  policy  manager  would  provide  a  spectrum  of  adaptive  actions 
from  advisories  to  cautions  to  warnings  to  auto  heading/alt  changes  based  on  flight  paths  past  the 
minimum  allowable  approach  range  toward  threats.  This  automation  manager  would  consider  the 
use  of  terrain  and  range  line  of  sight  effects  in  making  an  aspect/course/altitude  change  input;  the 
signature  policy  manager  would  (in  the  proposed  integrated  system)  make  inputs  in  favor  of  or 
against  course  changes  (whether  automated  or  manual)  to  ensure  that  requested  courses  would  not 
inadvertently  generate  a  fatal  shot  solution  from  an  enemy  missile  site.  Yet  obviously,  some  third 
party  agent  is  necessary  to  perform  the  rapid,  multivariate  comparison  and  arbitration  tasks 
between  all  these  agents,  if  a  human  cannot  possibly  interpolate  and  calculate  quickly  enough. 

The  Need  for  an  Executive  Agent 

While  separately,  individual  automation  elements  may  be  useful,  the  emergence  of  far  more 
complex  combat  requirements  requires  users  to  interpolate  and  integrate  the  many  information 
variables  (such  as  signature,  envelope,  and  fuel  as  well  as  datalinks  and  weapons  control)  for 
multiple  controlled  UAVs,  during  multiple  weapon  engagements  with  hostile  moving  targets. 
USAF  Colonel  John  Boyd,  father  of  the  Observe,  Orient,  Decide,  and  Act  (OODA)  loop  model 
of  tactical  engagement,  noted  that  the  key  to  combat  aircraft  survival  and  autonomy  is  the  ability 
to  adapt  to  change  rapidly  and  to  capitalize  on  calculated  advantages  faster  than  one’s  opponent 
-  to  “get  within  the  enemy’s  OODA  loop”  (Boyd,  1976).  With  such  a  varied  range  of  automated 
policy  managers,  conflict  arbitration  via  human  or  automated  means  is  necessary.  Because  a 
single  human  cannot  meet  the  analytical  and  computational  requirement  to  comparatively 
perform  the  cross  application  functions  for  multiple  UAVs  within  a  tactically  significant  timeline 
for  multiple  controlled  vehicles,  the  GCS  must  be  equipped  with  an  overarching  Executive 
Agent. 

Such  an  Executive  would  constantly  monitor  the  individual  policy  managers  for  each  UAV  and 
adjudicate  recommended  automated  actions  based  on  preplanned  algorithmic  responses  for  most 
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cases;  the  Executive  would  both  provide  more  urgent  advisories  (would  inform,  then  prompt, 
then  warn)  to  cue  user  intervention  based  on  the  severity  of  impact  of  the  problem  within  a 
tactically  significant  timeline  (e.g.  the  UAV  is  headed  for  a  threat,  turn  the  UAV  to  avoid 
detection,  and  finally  maneuver  the  UAV  to  defeat  an  engagement).  In  Boyd’s  terms,  the  control 
loop  authority  (human  or  Executive)  must  perform  general-to-specific  reasoning  -  deduction, 
analysis,  and  differentiation,  while  also  performing  specific -to  general  reasoning  related  to 
induction,  synthesis,  and  integration  tasks  (Boyd,  1976). 

In  most  cases,  the  Executive  would  employ  hierarchical  weightings  to  arbitrate  between 
conflicting  policy  managers  to  prioritize  actions  emphasizing  one  mission  aspect  over  another 
(such  as  a  prioritizing  lack  of  UAV  detection  over  choosing  the  most  fuel-efficient  return  route). 
In  all  cases.  Executive  arbitration  of  the  policy  managers  would  follow  mission  constraint 
settings  selected  during  mission  planning  by  the  user  (even  if  only  for  default  settings)  and 
consent  for  key  tasks  (e.g.  weapons  free  status  within  approved  engagement  constraints)  would 
necessarily  be  required. 

Conclusion 

By  equipping  proposed  future  multiple  combat  UAV  controlling  systems  with  agile.  Executive 
level  controllers  which  can  rapidly  perform  multivariate,  weighted,  arbitrations,  time  critical 
combat  tasks  be  met  within  the  multiple  UAV  control  paradigm.  Significant  further  mission  task 
analysis  and  requirements  decomposition  is  necessary  to  ensure  that  further  platform  specific  top 
level  and  detailed  level  design  requirements  are  properly  decomposed  and  allocated. 
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Outline 

Year  I  deliverables  summary 
Guidance 

Software  Failure  and  Reliability  Assessment  Tool 
(SFRAT) 

-  Architecture 

-  Review  of  Year  I  functionality 

-  Y ear  II  functionality 

Software  Defect  Estimation  Tool  (SweET) 

Goals 
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State  of  software  reliability 

Software  reliability  studied  for  50+  years 

-  Methods  have  not  gained  widespread  use 

•  Disconnect  between  research  and  practice 

Diverse  set  of  stakeholders 

-  Reliability  engineers 

•  May  lack  software  development  experience 

-  Software  engineers 

•  May  be  unfamiliar  with  methods  to  predict  software  reliability 
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YEAR  I  (3/15-2/16) 
DELIVERABLE  SUMMARY 
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Summary  of  Year  I  deliverables 

Implemented  open  source  software  reliability  tool 

-  Data  conversion  routines 

-  Trend  tests  for  reliability  growth 

-  Two  failure  rate  models 

•  Assume  failure  rate  decreases  as  faults  detected  and  removed 

-  Three  failure  count  models 

•  Count  faults  detected  as  function  of  time 

-  Tested  on  dozens  of  data  sets 

-  Two  goodness  of  fit  measures 
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Estimates  enabled  by  software 

reliability  models 

Number  of 

-  Faults  detected  with  additional  testing 

-  Remaining  faults 

Mean  time  to  failure  (MTTF)  of  next  fault 

-  Testing  time  needed  to  remove  next  k  faults 

Probability  software  does  not  fail  before 
completion  of  fixed  duration  mission 


Failure  rate  (F/ms) 
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Failure  rate  model 


Model  characterizes  decreasing  trend  in 

failure  rate 


Failure  count 
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Failure  time/count  models 


Model  characterizes  fault  discovery  process 
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sasdlc.org/lab/projects/srt.html 


Software  Intensive  Research  Laboratory  curriculum  vitae  Teachirg  Research  students  Fun 


Software  Failure  and  Reliability  Assessment  Tool  (SFRAT) 


Description 

The  key  to  the  success  of  all  software  is  its  reliability.  The  Softw^are  Failure  and  Reliability  Assessment  Tool  (SFRAT)  is  an  open  source  application  to  estimate  and  predict  the  reliability  of  a 
software  system  during  test  and  operation.  It  allow's  users  to  answer  the  following  questions  about  a  software  system  during  test; 

1.  Is  the  software  read>^  to  release  (has  it  achieved  a  specified  reliability  goal)? 

2.  How  much  more  time  and  test  effort  will  be  required  to  achieve  a  specified  goal? 

3.  What  will  be  the  consequences  to  the  system's  operational  reliability  if  not  enough  testing  resources  are  available? 

SFRAT  runs  under  the  R  statistical  programming  framework  and  can  be  used  on  computers  running  Windows,  Mac  OS  X,  or  Linux 

Resources 

WARNING;  Web  instance  is  for  demonstration  only.  Please  do  not  upload  sensitive  data  to  the  site 

Web  instance 
Example  failure  data  sets 
SFRAT  Github  repository 
User's  Guide 
Contributor's  Guide 

Publications 

Search:  I 


Year  Type 


Publication 
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Software  Reliability  Growth  Modeling 

•  No  single  model  characterizes  all  data  sets  best 

•  Models  supplementary  mathematical  guidepost 

-  Used  in  conjunction  with  SDLC  activities  to  identify, 
implement,  and  test  functional  requirements 

•  Do  not  prescribe  a  single  model 

•  Learn  to  track  before  planning  in  SEPs  &  TEMPs 

•  Emphasize 

-  Effective  communication  between  system,  reliability, 
and  software  engineers 

-  Frequent  use  of  quantitative  SRGM  throughout  DT  and 
OT  to  assess  progress  toward  software  and  system 
reliabilitv  goals 
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Software  Reliability  Growth  Tracking 

•  For  reliability  growth  tracking  to  be  effective 

-  Failures  and  their  severity  must  be  clearly  defined 

-  Impact  on  mission  and  end-to-end  capability  in  order  to 
produce  data  suitable  for  reliability  growth  tracking 

-  Will  be  impacted  by  updates  to  interacting  subsystems 
including  hardware,  mechanical,  sensing,  and  operator 
usage 
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Data  formats 

Based  on  data  formats 

-  Failure  Rate  models 

•  Inter-failure  times  -  time  between  (i  —  1)^^  and  i 
failure,  defined  as  =  (T^  — 

•  Failure  times  -  vector  of  failure  times, 

T  ti,  t-2_i  ■■■  t  tjx  ^ 

-  Failure  Counting  models 

•  Failure  count  data  -  length  of  the  interval  and 
number  failures  observed  within  it, 

<  T,  K  >=<  (ti,  /Ci),  (t2,  ^2), ... ,  (tn-  ^n)  > 

-  Possible  to  use  change  requests  during  DT 
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Data  quality 

Accuracy 

-  Critically  depends  on  availability  of  failure  data 

-  Inaccurate  records  of  time  make  model  fitting  and 
prediction  difficult 

Even  when  data  available 

-  Practitioner  must  know  how  to  filter  and  organize  data 
for  use  in  models 

•  Filter  to  exclude:  non-software  issues,  duplicate  failures,  etc. . . 
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SOFTWARE  FAILURE  AND 
RELIABILITY  ASSESSMENT 
TOOL  (SFRAT) 
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ARCHITECTURE 
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SFRAT  user  modes 

Graphical  user  interface 

-  Web  and  intranet 

Developer  mode 

-  Incorporate  additional  models 

Power  user 

-  Incorporate  into  internal  software  testing  processes 

Benefits 

-  Can  help  contractors,  FFRDCs,  and  government 
quantitatively  assess  software  as  part  of  data  collection, 
reporting,  and  oversight 


UMass 


Dartmouth  university  of  Massachusetts  Dartmouth 


SFRAT  -  File  structure 


futility 


&  Data 

a.Data_Tools.R 

&  Metrics 
a.GORR 
I©  Plots 

a. PlotModelResults  .R 

b. Plot_Raw_Data.R 

c  .Plot_Trend_Tests  .R 

&  Prediction 

a.Detailed_prediction.R 

^  tables 

a.DataAndXrendXables.R 
^  b.ModelResultXable.R 

^  RunModels.R 


1^  trend_tests 

1 .  Laplace_trend_test.R 

2.  RAA.R 


models 


7  GO 
7  DSS 
7  Wei 
n  JM 
n  GM 


New  models  added  in  the  ''models”  folder 
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Power  user  mode 

Code  can  be  tailored  for  internal  use 

-  Build  into  existing  automated  software  testing 
procedures  to  provide  near  real-time  feedback 
of  reliability  trends 

-  Many  industry  standard  programming 
languages  can  call  R  functions 

•  Visual  Basic,  Java,  C/C#/C++,  and  Fortran 

•  Ensures  tool  will  integrate  smoothly 
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REVIEW  OF  YEAR  I  FUNCTIONALITY 
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SFRAT  -  Tab  view 

Softwar©  R©liability  Ass6ssm©nt  in  R  Select,  Analyze,  and  Filter  Data  Set  Up  and  Apply  Models  Query  Model  Results  Evaluate  Models 


Plot  Data  or  Trend  Test? 

(i)  Data  o  Trend  test 

Does  data  show  reliability  growth? 


Laplace  Test 


Specify  the  confidence  level  for  the  Laplace  Test 

0.9 


Choose  the  type  of  file  to  save  plots.  Tables  are  saved  as  CSV  files, 
(i)  JPEG  O  PDF  O  PNG  O  TIFF 

^  Save  Display 


Subset  the  failure  data  by  data  range 
Specify  the  data  range  to  which  models  will  be  applied. 

0 

^ - 

1_ 2_ 3 


0 
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Tab  1 

Select,  Analyze,  and  Filter  data 
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Tab  1  -  After  data  upload 


Select,  Analyze,  and  Subset  Failure  Data 

Specify  the  input  file  format 


Plot  Data  and  Trend  Test  Table 


Cumulative  Failures  vs.  Cumulative  Test  Time  of  SYS1 


•  Excel  (.xlsx)  CSV  (.csv) 
Select  a  failure  data  file 
I  Choose  File  |  model_data.xlsx 


Upload  complete 


Choose  Sheet 


SYS1 

- 

Choose  a  view  of  the  failure  data. 

Cumulative  Failures] 

- 

Times  Between  Failures 

Cumulative  Failures 

Failure  Intensity 

100- 

Plot  Data  or  Trend  Test? 


•  Data  Trend  test 

Does  data  show  reliability  growth? 


Laplace  Test 

Specify  the  confidence  level  for  the  Laplace  Test 

0.9 

Choose  the  type  of  file  to  save  plots.  Tables  are  saved  as  CSV  files. 

JPEG  PDF  •  PNG  TIFF 
i.  Save  Display 

Subset  the  failure  data  by  data  range 

Specify  the  data  range  to  which  models  will  be  applied. 


Cumulative  failure  data  view 
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Laplace  trend  test  -  SYSl  data 

Laplace  trend  test  of  SYS1 


Decreasing  trend  indicates  reliability  growth 
(Indicates  application  of  SRGM  appropriate) 


Laplace  Test  Statistic 
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Laplace  trend  test  -  J4  data 

Laplace  trend  test  of  J4 


-4- 


6  50  100  150 

Failure  Number 


Does  not  exhibit  reliability  growth 
(Indicates  additional  testing  required) 
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Running  Arithmetic  Average  - 


SYS  1  data 

Running  Average  trend  test  of  SYS1 


Increasing  trend  indicates  reliability  growth 
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Tab  2 

Set  Up  and  Apply  Models 


Cumulative  Failures 
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Cumulative  failures 

Cumulative  Failures  vs.  Cumulative  Test  Time  for  SYS1 


..  .  I  — Data  —  Geometric  Jelinski-Moranda 

Model  ..  Qeiayed  S-Shape  -  -  Goel-Okumoto  -  Weibull 

Plot  enables  comparison  of  data  and  model  fits 
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Time  between  failures 


Interfailure  Times  vs.  Cumulative  Test  Time  for  SYS1 


....  —  Data  --  Geometric  Jelinski-Moranda 

Model  Delayed  S-Shape  Goel-Okumoto  -  Weibull 

Times  between  failures  should  increase  (indicates  reliability  growth) 
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Failure  intensity 


0.0075- 


(/) 

c 

0.0050- 

_c 

CD 

‘c5 


0.0025- 


0.0000- 


Failure  Intensity  vs.  Cumulative  Test  Time  for  SYS1 
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0  30000  60000  90000 

Cumulative  Test  Time 


—  Data  --Geometric  Jelinski-Moranda 

Model  __  Qeiayed  S-Shape  Goel-Okumoto  -  Weibull 

Failure  intensity  should  decrease  (indicates  reliability  growth) 


Reliability  Growth 
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Reliability  growth  curve 

Reliability  Growth  vs.  Cumulative  Test  Time  for  SYS1 :  Operational  Time  of  600 
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Failure  Predictions 
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Evaluate  Models 
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AIC  and  PSSE 
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Upper  and  lower  confidence  limits 

-  Graphical  and  tabular  values 

Model  Evaluation  Criteria 

-  Prequential  likelihood  (PL)  ratio 

•  Identify  model  more  likely  to  produce  accurate 
estimates 

-  Higher  preferred 

-  Model  bias  (MB)  and  MB  trend 

•  Indicate  whether  model  over/underestimates 
times  between  failures 

Optimal  release 
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Configure  and  Apply  Models 

Specify  the  number  of  failures  for  which  the 
models  will  make  predictions 
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estimation  interval. 
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estimates. 
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models  will  predict 


Choose  one  or  more  models  to  run,  or  exclude  one  or 
more  models. 
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Jelinski-Moranda 


Run  Selected  Models 


Display  Model  Results 


Model  Result  Plot  Model  Result  Table 
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bounds.  ^ 


Cumulative  Failures  vs.  Cumulative  Test  Time  for  SYS1 


Choose  one  or  more  sets  of  model  results  to  display. 
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Select,  Analyze,  and  Filter  Data  Set  Up  and  Apply  Models  Query  Model  Results  Evaluate  Models 


Evaluate  Model  goodness  of  fit 
and  Applicability 

Choose  one  or  more  models  for  which  the 
results  will  be  evaluated. 

Choose  one  or  more  sets  of  model  results 

Geometric  Goel-Okumoto 
Jelinski-Moranda 

Specify  the  Percent  Data  for  PSSE 
0.9  0 

Draw  the  plot  with  data  points  and  lines,  points 
only,  or  lines  only? 

O  Both  O  Points  O  Lines 

Save  model  evaluations  as  PDF  or  CSV? 
e  CSV  O  PDF 

^  Save  Model  Evaluations 


Evaluation  Summary  Model  Evaluation  Plot  Model  Evaluation  Table 
Choose  a  model  applicability  evaluation  to  plot. 


Prequential  Likelihood  Ratio 


Prequential  Likelihood  Ratio  vs.  Failure  Number  for  SYS1 


Model  Geometric  Goel-Okumoto  Jelinski-Moranda 
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Evaluate  Model  goodness  of  fit 
and  Applicability 


Choose  one  or  more  models  for  which  the 
results  will  be  evaluated. 

Choose  one  or  more  sets  of  model  results 

Geometric  Goel-Okumoto 
Jelinski-Moranda 

Specify  the  Percent  Data  for  PSSE 
0.9  : 


Draw  the  plot  with  data  points  and  lines,  points 
only,  or  lines  only? 

O  Both  O  Points  O  Lines 

Save  model  evaluations  as  PDF  or  CSV? 

O  CSV  _  PDF 

±  Save  Model  Evaluations 


Evaluation  Summary  Model  Evaluation  Plot  Model  Evaluation  Table 
Choose  a  model  applicability  evaluation  to  plot. 


Model  Bias 


Models  above  line  estimate  more  frequent  times  between  failures  than  those  observed 
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Evaluate  Model  goodness  of  fit 
and  Applicability 


Choose  one  or  more  models  for  which  the 
results  will  be  evaluated. 

Choose  one  or  more  sets  of  model  results 

Geometric  Goel-Okumoto 
Jelinski-Moranda 


Specify  the  Percent  Data  for  PSSE 
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Draw  the  plot  with  data  points  and  lines,  points 
only,  or  lines  only? 

O  Both  O  Points  O  Lines 

Save  model  evaluations  as  PDF  or  CSV? 

O  CSV  O  PDF 
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Evaluation  Summary  Model  Evaluation  Plot  Model  Evaluation  Table 
Choose  a  model  applicability  evaluation  to  plot 


Model  Bias  Trend 


Model  Bias  Trend  (y-plot)  for  SYS1 


Models  below  line  estimate  more  frequent  times  between  failures  than  those  observed 
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SOFTWARE  DEFECT 
ESTIMATION  TOOL  (SWEET) 
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SWEEP  (Software  Error 
Estimation  Program) 

Implemented  four  modes 

1 .  Time-based  model 

•  Estimates  and  tracks  errors  during  system  test  and  integration 
cycle 

2.  Phase-based  model 

•  Provides  defect  information  before  running  any  code 

3 .  Planning  aid 

•  Generates  an  error  discovery  profile  based  on  historical  data 

4.  Defect  injection  model 

•  Allows  user  to  understand  probable  defect  injection  profile 
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Software  Intensive  Research  Laboratory 


Curriculum  Vitae  Teaching  Research  Students  Fun 


Software  Defect  Estimation  Tooi  (SweET) 


Description 

The  Software  Defect  Estimation  Tool  (SweET)  is  an  open  source  application  to  track  error  identification  and  removal  efforts  during  the  software  development  lifecycle.  SwEET  is  a  free  and 
open  source  version  of  the  Software  Error  Estimation  Program  (SWEEP)  and  SweET  uses  Weibull  software  reliability  growth  model  utilizing  Expectation  Conditional  Maximization  algorithm 
to  ensure  stability  and  performance  of  the  model  fitting  process.  SweET  simplifies  four  models  of  SWEEP  into  three  modes: 

1.  Mode  A:  Time-based  model:  Estimates  and  tracks  errors  during  system  test  and  integration  cycles. 

2.  Mode  B:  Phase-based  and  planning  aid  model:  Predict  and  track  defects  for  multiple  phases  and  can  provide  defect  information  before  running  any  code,  whereas  the  planning  aid 
model  generates  an  error  discovery  profile  based  on  the  phase  based  historical  data  to  help  a  software  prohect  achieve  its  objectives. 

3.  Mode  C:  Defect  injection  model:  Allows  the  user  to  understand  the  probable  defect  injection  profile  and  resulting  efficiency  and  effectiveness  of  the  verification  process. 

SweET  runs  under  the  Python  3.x  programming  framework  and  can  be  used  on  computers  running  Windows,  Mac  OS  X,  or  Linux 

Resources 

Example  data  sets 
SweET  Github  repository 
User's  Guide  (In  preparation) 
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GOALS 
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Activities 

Update  documentation 

Outreach,  education,  and  training 

-  Visit  DoD  labs  and  listen  to  practical  concerns 
underlying  modeling  requirements 

-  Work  with  existing  users 

Coordinate  contributions  from  developers 

-  Failure  severity  decomposition 

-  Software  readiness  metrics 

-  Additional  models,  Bayesian,  covariate 

-  Expand  architecture  to  additional  stages  of  lifecycle 
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Covariate  data  example 
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Covariate  model  data  fit 
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Stakeholder  outreach 
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Sea  Lanes  Remain  the  Lifeblood  of  Our  Economy 


Competition  is  Back 
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Changing  Environment 


Short  Range 
Ballistic  Missiles 


AAAAAAAAA 
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11111111 J 


Advanced  Fighter/  Strike 
Aircraft  (3'^^/4*'^  Generation) 


Advanced  ASCM  Capable 
Surface  Combatants  (DD/FF) 


Aircraft  Carrier 


PRC  Military  Spending  (Official  Figures) 


120 


Spending  in  Billions  (US) 


1999  2001  2003  2005  2007  2009  2011  2013 


2014  Budget  Increased  12.2% 


1999 
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Advanced  Attack 
Submarines 


2015 


USN  and  PLA(N)  Capability  Fielding  Trends 
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CNO’s  Challenges  to  all  Flag/SES 


5  Key  Points 


Must  be  competitive  ^  Existentiai  Threat  ^  No  #2 
Think  Strategicaiiy  ^  Criticai  Thinking 


Going  Digitai 

Outcome  /  Product  Oriented  ^  Vice  Process 
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NAVAIR  Response 


strategic  Initiatives  -  Focus  on  Speed 


•  Capabilities  Based  Acquisition  -  Rapid  delivery  of  integrated  capabilities 

•  Sustainment  Vision  2020  -  Predictive,  integrated  sustainment  operations 

•  Digital  Business  Operations  -  Integrated  business  systems  "apps"  at  the  desktop 


Accelerating  delivery  of  fully  integrated  capabilities  which  are  designed, 
developed,  and  sustained  in  a  Model  Based  Digital  Environment 

IS  a  Systems  Engineering  challenge 
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Capabilities  Based  Acquisition 

Digital  Thread  Enables  Rapid  Delivery  of  Integrated  Capabilities 


I 


I 


Integrated  Warfare  Analysis 
establishes  CONEMPS, 
Effects-Chains,  required  attributes 


SoS  Model 


CONEMPS  and  Effects  Chains  are 
modeled  at  the  System  of  Systems 
(SoS)  level 


System  models  form  “Constructive” 
basis  for  LVC  M&S  environment 


Systems  are  developed  in  a 
Model-Based  environment 
(SE  Transformation) 


I  El 


Digital  Linkage 


Enabling  Capabiiities-Based  T&E 


Constructive  Virtual  Live 
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SET  Framework 


Elimination  of  paper  CDRL  artifacts  and 
large-scaie  design  reviews 


Continuous  insight/oversight  via  digitai 
coliaborative  environment  and  interaction  with  the 

Singie  Source  of  Truth 
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Multi-Disciplinary  Analysis  &  Optimization 
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Execution  Framework 
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Surrogate  System  Experiment 


•  Simulate  Execution  of  SET  Framework 

•  Use  UAV  scenario  developed  in 
SERC  models 

-  Combine  SysML  models  already  in 
development  -  requirements, 
with  functional  and  logical  views 

-  Use  MDAO  of  parametrics  for  some  KPPs 

•  Consider  NATO  example 

•  Characterize  objectives  and  thresholds 

-  Create  a  model-based  contract  simulating  RFP  /  SOW 

•  Use  commercial  organization  to  simulate  industry  organization 

-  Refinement  of  SysML  models  to  reflect  corrections  /  innovations  with  physical  allocation 
views 

-  Integrate  with  multi-physics-based  Initial  Balanced  Design 

-  Simulate  continuous  virtual  reviews  and  derive  new  objective  measures  for  assessing 
maturing  design 

•  Simulate  source  selection  based  on  dynamic  models  and  simulations 


Elimination  of  paper  CDRL  artifacts  and 
large-scale  design  reviews 
Continuous  insight/oversight  via  digital 
collaborative  environment  and 
interaction  with  the  Single  Source  of 
Truth 


oyvivin  opev  m  j 

_  » e»h».K.- 

Mochanical  Dosign  Modols 

it 

Electrical  Design  Models 

Ifel 

SoftMare  Design  Models 

t  .mB  ' 

Testing  Methods  &  Models 

Analysis  Tools 

MDAOVSET-BASED  DESIGN 


*  Muki-Discipfinary  Analysis  &  Optimization 
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Industry-Government  Partnership 


•  SET  applies  to  both  Government  and  Industry 

•  Government  must  reassess  its  role  in  the  acquisition 
process  and  the  methods  for  executing  that  role 

1 .  Criteria  for  gov’t  involvement  /  oversight  (not  every  decision) 

2.  If  involved,  must  be  on  developer’s  timeline 

3.  Must  bring  value  to  the  decision  -  not  just  positional  authority 

•  Industry  must  fully  leverage  advances  in  HPC-enabled 
models  and  participate  in  establishing  a  collaborative, 
integrated  digital  environment  which 
enables  continuous  interaction 
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For  More  Information,  Contact: 


Mr.  Dave  Cohen,  Director  Systems  Engineering 

(301)757-5542 

david.cohen@navv.mii 
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Capabilities  Based  Acquisition 


Integrated 
Warfare  Analysis 

Fleet  operators  work 
with  engineering  to  develop 
concepts  of  employment 
(CONEMPS),  effects-chains 
&  required  attributes 


CONEMPS  and 
effects-chains  are 
modeled  at  the 
System  of  Systems 
(SoS)  level 


System  Model 

Systems  are  developed 
in  a  model-based  integrated 
digital  environment 
(SE  Transformation) 
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System  models  form 
Constructive" 
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Required  for  Fleet  Integrated  Ops 


Integrated  Digital  Environment  acceierates  deiivery  of  operationaliy  relevant  capabilities 
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Sustainment  Vision  2020  -  What  it  Looks  Like 


RAW  DATA 


APPLICATIONS 
/  TOOLS 


ANALYSIS 


FLEET  DECISIONS 
FLEET  SUPPORT 


STATUS-TRENDS-PREDICTIONS 

IS 


Naval  Aviation  Sustainment  Center 


Universal  Information 
Faster  Decision  Making 
Predictive  Sustainment  Planning 
Reduced  Cost 
Increased  Readiness 


as  Optimization  and 
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Resources  to  Meet 
Fleet  Needs... 
Maintenance  Planning 
Supply  Support 
Workforce 
Facilities 


19 


Distribution  Statement  A: 
Approved  for  public  release;  distribution  unlimited  PR#  201 7-928 


Data 


Digital  Transformation:  Business  Operations 


I 


Today:  Monoliths  in  Silos 


Tomorrow:  The  Composable  Business 


I 


Applications  built  in  silos 
Data  duplicated  in  tools  causing  manual  re-entry 
Data  locked  in  tools  preventing  ease  of  re-use 
Application  changes  slow  and  costly 

Functionality  duplicated  across  tools  causing  inconsistencies  and 
difficulty  in  coordinating  business  process  changes 
Can’t  tailor  to  support  unique  process  across  business  units 
Similar  functionality  reinvented  over  and  over  again  at  great  cost 


Infrastructure  enables  tailored  applications  while  maintaining  consistent  core 
business  rules  and  data 

Applications  “composed”  from  reusable  services 

Consistency  of  Data  and  Business  Rules  across  Business  Operations 

Agility  in  supporting  rapid  Business  Process  changes 

Lightweight  services  with  short  development  lifecycle 

Individual  Services  “owned”  by  Authoritative  Competency 
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USN  vs  PLA(N)  Capability  Fielding 


We’re  Being  Out-Sticked 


USN  Warfighting  Advantage  Against  PLA(N)  has  Steadily  Eroded 
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DHS  SCIENCE  AND  TECHNOLOGY 


DHS  Systems  Engineering  Acquisition 
Challenges  and  Issues 

NDIA  20**^  Annual  National  SE  Conference 
October  25,  2017 
James  D.  Tuttle 

Chief  Systems  Engineer 
Science  and  Technology  Directorate 
Department  Homeland  Security 


Homeland 
^  Security 

Science  and  Technology 


Major  DHS  Operating 
Components 


•  Transportation  Security  Administration  (TSA) 

•  U.S.  Coast  Guard  (USGC) 

•  U.S.  Secret  Service  (USSS) 

•  U.S.  Customs  and  Border  Protection  (CBP) 

•  U.S.  Citizenship  and  Immigration  Services  (CIS) 

•  U.S.  Immigration  and  Customs  Enforcement  (ICE) 

•  Federal  Emergency  Management  Agency  (FEMA) 

•  Domestic  Nuciear  Detection  Office  (DNDO) 


DHS  Science  and  Technology  Directorate  |  DHS  Systems  Engineering  Acquisition  Challenges  and  Issues 


TSA  Programs 


Electronic  Baggage  Screening  Program 


Passenger  Screening  Program 


F 
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USCG  Air  Programs 


C27-J 


HC-144 


HC-130J 


MH-60J  HH-65 
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USCG  Surface  Programs 


Fast  Response  Cutter 


National  Security  Cutter 


Offshore  Patrol  Cutter 


Heavy  Polar  Icebreaker 
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ICE  Programs. 
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Student  and  Exchange  Visitor  Information  System 
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Student  &  Exchange  Visitor 
Information  System 

1-800-892-4829 

SEVIS  Help  Desk 
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Search  *  New  Student 


Listing  of  Schools 

Student  Lists  Reports  Downloads 


SEVPSC nool  for  Adv»Ktfd  SEVIS  Studies  WAS214F07070000  v| 


Select  a  school  to  see  a  kst  or  campuses  for  that  school 


^  SEVP  School  for  Advanced  $eyis_$tgdies  WAS214F07070000 

Select  Name  of  Campus  Campus  Code 

iei  Mam  Alpha  Campus  WAS214F07070000 

O  SEVP  Fairfax  Second  School  WAS214F07070001 


Location  (City.  State) 
Fairfax.  VA 
Fairfax.  VA 


Role 

OSO 

OSO 


®9  mtScaiM  an  alert  tor  mat  campus 

'Ar  indcaltsmal  the  POSO  of  toe  man  campus  has  to  apply  tor  recarMcabon 

9  toocatesmal  toe  abrwy  of  the  schoorsoAciais  to  create  iww  student  records  has  been  osabied  by  SEVP 

^  imacaiestoattheKhoorsabaiiy  to  accept  transser-n  student  records  has  been  deabied  by  SEVP 

indKaiestoat  toe  abMy  of  toe  schoorsoAciais  to  transfer  out  student  records  has  been  disabled  by  SEVP 
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FEMA  Programs 


Integrated  Public  Alert  and 
Warning  System 


GMM  Target  Solution 


Simplify  the 


User-centered, 

business-driven 

s^pproach 


Timeliness 
of  funding 


Reduce 
sustainment 
I  costs  ^ 


Business 

performance 


Compliance 


Business 
inteliigefKe  and 
decision  making^ 


Grants 
Management 
Modernization 


Allocation  of 
grant  funds 


National  Flood 
Insurance  Program 


Logistics  Supply  Chain 
Management  System 
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USCIS  Programs 


Transformation  Program 


US. Citizenship  QIaire  Supported  the  Goals  of  ^ 

USCIS  Transformation  Program 


Program  Performance 

Supported  the  review  of  QASP  standards  and  Measurement  Planning 
Supported  the  Program  Performance  goals  by  reporting  QASP  Metrics 
Quality  Assurance 

V  Program  Product  and  Process  Quality  to  meet  CMMI  Level-3  maturity 
«  Led  Product  and  Managed  Program  Quality  and  Accessibility  testing  (Section  508  compliance) 
throughout  ail  Lifecycle  Phases  (Iterative  and  Agile) 

^  Managed  Process  Audits,  Facilitated  Lessons  Learned  Sessions,  Lead  Root  Cause  Analysis,  and 
Managed  Program  Risk 
Water  fall  to  Agile  Transition 

Tailored  Agile  Lifecycle,  and  Quality  Review  Methods  to  meet  DHS  SELC  standards 
Systems  Operations  and  Maintenance 
♦  Streamlined  Internal  User  Provisioning  Processes 

Demonstrated  improved  efficiencies  and  timely  User  Provisioning 
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CBP  Programs 


Border  Wall 


Video  Surveillance  Systems 


argo  Processing  and  Inspection 


ij'sui'nimw.! 
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SE  Challenges  In  DHS 


-  Solutions  encompass  the  entire  Homeland  Security  Enterprise  (HSE) 

-  Diverse  customer  base,  with  different  rules,  restrictions,  and  users 

-  Diverse  “mission  set”  including  security,  immigration,  trade/commerce, 
disaster  planning,  and  response 

-  Significant  IT  and  Embedded/Mixed  IT  Solutions 

-  Privacy  and  security  concerns  related  to  sharing  data 

-  Controlling  proprietary,  business,  or  law  enforcement-sensitive  data 

-  Flexible  and  resilient  solutions  to  respond  to  emergent  threats  and 
national  disasters 

-  Procurement  vs.  acquisition  mentality  in  many  parts  of  the  HSE 

-  Too  much  focus  on  compliance/templates  vice  critical  analysis 
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SE  Staffing  Challenges 


•  DHS  has  a  Level  I,  II,  III  SE  Acquisition  Certification  program  and 
activity  instructing  courses 

•  Inconsistency/limited  participation 

•  No  consistent  SE  staffing  within  Acquisition  programs  and 
Component  Acquisition  oversight  offices 

•  Secretary  directed  all  major  acquisition  programs  and  Component 
Acquisition  oversight  offices  to  resource  SE  expertise 

•  Initiative  established  in  Secretary’s  FY19-23  Resource  Planning 
Guidance  (for  Components  developing  Resource  Allocation  Plans) 
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SE  Challenges  in 

DHS  Acquisition  Programs 


Weak  Solution  Analysis/Analysis  of  Alternatives  prior  to  approval 

AoAs  have  not  been  properly  scoped  and  executed 

•  Frequently  focused  on  all  COTS,  no  COTS,  or  a  mix,  vice  analyzing  the 
operational  and  technical  solution  space 

•  Poor  definition  of  alternatives  and  evaluation  criteria 

•  Relative  ranking  of  alternatives  among  each  other  vice  against  actual  mission 
need/requirements 

•  AoAs  often  not  informed  by  data  from  pilots/prototypes/testing 

•  AoAs  as  a  process  to  document  the  acquisition,  vice  analyze,  learn,  and 
modify  to  enable  better  decisions 
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SE  Challenges  In 

DHS  Acquisition  Programs 


Poorly  developed  Operational  Requirements  and  CONORS 

•  Limited  CONORS  scope 

•  Scenarios  do  not  describe  the  complete  operation  or  even  all  the  tasks  the 
proposed  solution  must  perform  (only  small  mission  tasks  that  need  improvement) 

•  Boundaries,  interfaces,  key  external  stakeholders/systems  of  the  proposed  solution 
not  clearly  defined/understood 

•  Limited  analysis  leading  to  the  actual  Operational  Requirement 

•  Often  not  operationally  focused 

•  Often  focused  on  “user  needs”  that  reflect  specific  problems 

•  Limited  analysis  to  support  Threshold  and  Objective  performance  values 
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SE  Oversight  Challenges 


•  HQ  oversight  has  heavily  focused  on  programmatic  oversight 

•  Focused  on  checklists  and  artifact  existence 

•  Limited  evaluation  of  the  quality/substance  of  artifacts 

•  In  2015,  Secretary  directed  S&T  to  conduct  Technical 
Assessments  on  major  acquisition  programs 

•  SE-based  Technical  Assessments  of  major  acquisition  programs 

•  Focused  on  quality,  not  quantity 

•  Not  if  a  program  has  operational  requirements;  are  requirements 

•  Feasible  Testable 

•  Clear  Backed  by  objective  analysis 

•  Complements  existing  acquisition  programmatic  oversight  processes 

•  Conducted  prior  to  commencement  of  design/development/integration 
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Technical  Assessment  Impacts 


•  Greater  up-front  analysis  =  technically  stronger  programs 

•  Threshold  and  objective  parameters  of  operational  requirements, 
backed  by  objective  analysis 

•  CONORS  that  better  describe  end-to-end  system  operation,  not  only 
parts  of  the  system 

•  Trade-off  analysis  during  the  Analysis  of  Alternatives  that 
informs/refines  operational  requirements  and  CONORS 

•  Holistic  (programmatic  and  technical)  perspective  for  Acquisition 
executives  before  design/development/integration  activities 

•  More  informed  decision-making  by  Acquisition  executives 
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Conclusion 


•  DHS  enterprise  has  a  wide  range  of  mission  areas  and  a  civilian/iaw 
enforcement  cuiture 

•  Acquisition  stiii  somewhat  synonymous  with  procurement 

•  DHS  reaiizes  SE  needs  to  be  institutionalized  across  the  Department 
and  is  making  headway 

•  Developed  rigorous  SELC  Guidance 

•  Implemented  Technical  Assessments  for  Major  Acquisition  Programs 

•  AoA,  ORDs,  CONORS,  etc.  improving  as  SEs  engage 

•  Looking  to  continue  coiiaboration  across  government  and  industry 
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AFLCMC/EZ 
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L0A2 
Goal  &  Objectives 

■  Goal:  Efficiently  and  effectively  incorporate  Systems  Security 
Engineering  (SSE)  into  the  Systems  Engineering  (SE)  process  in  aii 
phases  of  the  Acquisition  Lifecycie  to  increase  cyber  resiiience  in  AF 
systems 

■  Team  Members:  AFLCMC,  AFTC,  SMC,  NWC,  AFMC,  AFRL,  SMEs 

■  Objectives 

1.  Process  Integration:  Integrate  SSE  into  SE  processes  and 
deliverables 

2.  Process  Assessment:  Develop  metrics  to  measure  SSE  incorporation 
into  SE  processes  and  deliverables 

3.  Product  V  &  V:  Develop  system  cyber  test  and  evaluation  methodology 
and  capability  across  the  lifecycle  for  all  AF  systems  -  aircraft,  weapons, 
C4ISR,  IT,  Space,  Nuclear 
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Breaking  Barriers  ...  Since  1947 
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U.S.  AIR  FORCE 


LOA2 

Integrate  SSE  into  SE  Processes 


■  Status: 

■  Identified  OPRs  &  formalized  membership 

■  Implementing  Action  Plan 

■  Several  process  guides  drafted/in 
coordination 

■  SE  Tech  Review  entry/exit  criteria  drafted 

■  Cyber  scorecard  drafted;  pilot  apps  under 
way 

■  Cyber  Test  &  Evaluation  Study  Completed 


Risk  Assessment 


Enables  a  balanced  approach  for  delivering  capability  to  the  warfighter 


Systems  Engineering 


Systems 
Engineering 
Test  &  Evaluation 


1 


■  Near-term  Way  Ahead: 

■  Update  existing  guides  based  on  feedback  and  evolving 
policy/regulations 

■  Produce  deliverables  and  work  with  Cyber  Resiliency  -  Technical 
Advisory  Council  (CR-TAC)  to  disseminate/  institutionalize 

■  Continue  interfacing  across  LOAs,  especially  with  the  LOA  3  Cyber 
Resiliency  Support  Team  (CRST) 
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Breaking  Barriers  ...  Since  1947 


3 


LOA  2  Organization 


Overarching 
Andrew  Martin 


Champion 
Dr.  Ken  Barker 


Tech  Advisors 
William  Mejias 
Patrick  Imlay 


[  Task  1 .4 

[  Martin/Zimmerman  /  Ehlers  /  Salazar 


Objective  3  Lead 
Di^osept^ichol^^^H 

Task  3.2 

Hobin/Newton/Jones/ 

Camp 

^  Sean  Borror 

I _ No  Resources 


Task  1 .5 

Zimmerman  /  Jones  /  Ware  /  Salazar 
Task  1 .6 
Task  1 .7 

Bumgardner  /  Furey-Defely  /  Scruggs 
Johnson  Fiore/Jones/Alexander 


Legend 


Wright-Patt  ■  Los  Angeles  ■  Hanscom  ■  Edwards 


Kirtland  ■  Peterson 
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Objective  1: 
Process  Integration 


■  Objective  Description:  Integrate  SSE  principles  into  SE  processes 
and  deliverables 

■  OPRs: 

■  Leads:  Mr.  Nick  Shouse,  AFLCMC/EZS; 

Capt  Cameron  Barnes,  SMC/ENX 

■  Reps  from  AFLCMC,  SMC,  AFNWC,  AFMC,  FFRDCs,  Contractor 


SMEs 
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Breaking  Barriers  ...  Since  1947 
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U.S.  AIR  FORCE 
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Breaking  Barriers  ...  Since  1947 


LOA  2,  Task  1. 1 

Establish  executable  process  for  CPI  &  CC  ID 


Task  Description  &  Deliverables 


Resource  Loaded  Schedule 


Description 

•  Provide  process  guidance  that  enables  programs  to 
accurately  identify  and  obtain  independent  review 
and  validation  of  CPI/CC. 

Deliverables 

•  CPI  and  CC  Identification  Process  Guide 


FY16 

FY17 

FY18 

FY19 

FY20 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

? 


CPI/CC  ID 
Guide  V1 .0 


Updates  (as  necessary) 


Ware,  Zimmerman,  Martin 


Done  Criteria 

•  Approval  of  CPI/CC  Identification  Process  Guide 
by  CR-TAC 

•  CPI/CC  Identification  Process  Guide  submitted  for 
consideration  to  SAF/AQR  for  adoption  as  an  Air 
Force  Pamphlet  (AFPAM)  or  referenced  by  AFPAM 
63-113,  Program  Protection  Planning  for  Life  Cycle 
Management 

•  Guide  posted  to  site  accessible  by  all  acquisition 
center  program  offices 
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Institutionalization 

•  Targeted  Audience  -  Acquisition  center  program 
office,  especially  PMs,  SEs,  and  SSEs 

•  Training  -  Analyze  whether  existing  module  of 
Program  Protection  course  on  CPI/CC  ID  is  sufficient 

•  Accountability  -  Best  practice  to  ensure  correct 
implementation  of  DoDI  5200.39  and  5200.44 

•  Sustainment  organization  -  Transition  in  FY1 9  or  20 
to  AFLCMC/EZSP  and  SMC/ENX  for  sustainment 


Breaking  Barriers  ...  Since  1947 


LOA  2,  Task  1.2 

Define  SSE  &  Integrate  SSE  into  SE 


Task  Description  &  Deliverabies 

Descriotion 

•  To  provide  understanding  of  SSE  terms  and 
concepts  within  a  Guide  for  Accomplishing 
Comprehensive  SSE 

Deliverables 

•  Guide  for  Accomplishing  Comprehensive  SSE, 
including  Program  Work  Breakdown  Structure 
(WBS),  artifacts,  and  templates 

Resource  Loaded  Schedule 

FY16 

FY17 

FY18 

FY19 

FY20 

1 

2  3  4 

12  3  4 

1  2  3 

4 

12  3  4 

1 

2  3  4 

AFLCMC 

PPP 

standard 

Process 

WBS  &  ^ 
Artifacts 

USAF  ^ 

Comprehensive^ 
Guide  to  SSE 

Non-LOA  2  Support  (AFLCMC/EZSP) 

LOA  2  Support  (Zimmerman,  Martin,  Ware,  Moyer) 

Funded  support  increases  in  FY18 

Done  Criteria 

•  Approval  of  the  Guide  for  Accomplishing 
Comprehensive  SSE  by  CR-TAC 

•  Submitted  to  SAF/AQR  for  consideration  as  a 
replacement  for  the  existing  AFPAM  63-113 
{Program  Protection  Planning  for  Life  Cycle 
Management) 

•  Guide  posted  to  site  accessible  by  all  acquisition 
center  program  offices 

DISTRIBUTION  A.  Approved  for  public  release:  distribution  unlimited. 

Institutionalization 

•  Targeted  Audience  -  Acquisition  center  program 
office,  especially  PMs,  SEs,  and  SSEs 

•  Training  -  Potentially  add  module  to  Program 
Protection  course 

•  Accountability  -  Recommended  to  PEOs  as  a  best 
practice 

•  Sustainment  organization  -  Transition  in  FY20  to 
AFLCMC/EZSP  and  SMC/ENX  for  sustainment 

Breaking  Barriers  ...  Since  1947 


LOA  2,  Task  1.3 

Establish  executable  process  for  System  Security  Risk 

Management 


Task  Description  &  Deliverables 


Resource  Loaded  Schedule 


Description 

•  Provide  one  integrated  system  security  risk 
management  process  that  programs  execute  as 
part  of  their  overarching  risk  management  process, 
including  the  steps  for  risk  planning,  identifying, 
analyzing,  handling,  and  monitoring. 

Deliverables 

•  Risk  Management  Supplement  to  AFPAM  63-128, 
Integrated  Life  Cycle  Management  -  Supplemental 
guide  to  integrate  system  security  risk  management 

Done  Criteria 


FY16 

FY17 

FY18 

FY19 

FY20 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

Risk 

Management 

Supplement 


Updates  (as  necessary) 


Zimmerman,  Imlay,  Mclnnes,  Ware,  Skujins, 
Newton 


Institutionalization 


•  Approval  of  the  Risk  Management  Supplement  by 
the  CR-TAC 

•  Submitted  for  consideration  to  SAF/AQR  for 
update  of  the  AFPAM  63-128,  Integrated  Life 
Cycle  Management,  to  include  system  security 
risk  management 

•  Supplement  posted  to  site  accessible  by  all 
acquisition  center  program  offices 


•  Targeted  Audience  -  Acquisition  center  program 
office,  especially  PMs 

•  Training  -TBD 

•  Accountability  -  Recommended  to  PEOs  as  a  best 
practice 

•  Sustainment  organization  -  Transition  in  FY1 9  or  20 
to  AFLCMC/EZAS  and  SMC/ENX  for  sustainment 
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Breaking  Barriers  ...  Since  1947 


LOA  2,  Task  1.4 

Develop  and  execute  acquisition  language  guidance 


Task  Description  &  Deliverables 


Resource  Loaded  Schedule 


Description 

•  Provide  SSE-focused  guidance  to  program  offices 
for  use  in  various  acquisition  docs 

•  Offers  programs  a  common  starting  point 

Deliverables 

•  USAF  SSE  Acquisition  Guidebook  -  Iterative 
development  with  periodic  publication  of 
updated  versions 


Done  Criteria 


FY16 

FY17 

FY18 

FY19 

FY20 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

V1.0  V1.1  V1.2  V1.3  V1.4  V1.5  V1.6  V1.7 


Ehlers,  Ware,  Martin,  Zimmerman 


Institutionalization 


•  Interim  deliveries/updates  made  as  new 
information  becomes  available  from  other  Cyber 
Campaign  Plan  activities 

•  Approval  of  the  Final  USAF  SSE  Acquisition 
Guidebook  by  CR-TAC 

•  Guide  posted  to  site  accessible  by  all  acquisition 
center  program  offices 


•  Targeted  Audience  -  Acquisition  center  program 
office,  especially  PMs,  SEs,  SSEs,  and  Contracts 

•  Training  -TBD 

•  Accountability  -  Recommended  to  PEOs  as  a  best 
practice 

•  Sustainment  organization  -  Transition  in  FY20  to 
AFLCMC/EZSI  and  SMC/ENX  for  sustainment 


DISTRIBUTION  A.  Approved  for  public  release:  distribution  unlimited. 


Breaking  Barriers  ...  Since  1947 


LOA  2,  Task  1.5 

Establish  SETR  SSE  Entry  &  Exit  Criteria 


Task  Description  &  Deliverables 


Resource  Loaded  Schedule 


Description 

•  Establish  SETR  SSE  entry/exit  criteria  that  program 
offices  across  AFLCMC,  SMC,  and  AFNWC  can  use 
to  evaluate  the  design  maturity  of  programs  during 
various  SETR  activities. 

Deliverables 

•  Updated  SETR  SSE  Entry/Exit  Criteria/Tasks 
outlined  within  the  USAF  SSE  Acq  Guidebook 

•  Updated  SETR  Toolset  with  SSE  Entry/Exit  Criteria 


FY16 

FY17 

FY18 

FY19 

FY20 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

SETR  Tool  Update 
Acq  Lang  Guide  V1 .2 


Note: 

Inc1:  ASR,  SRR,  SFR 
Inc  2:  PDR,  CRR,  SVR 
Inc  3:  FCA,  PCA,  PRR 


SETR  Tool  Update 
Acq  Lang  Guide  V1.3 


SETR  Tool  Update 
Acq  Lang  Guide  V1.4 


Skujins,  Martin, 
Boardman,  Jones,  Ware, 
Dailey,  Shealev 


Done  Criteria 


Institutionalization 


•  Final  SETR  Entry/Exit  Criteria  reviewed/approved 
by  the  CR-TAC 

•  Update  of  AFLCMC  SETR  Toolset  with  approval  by 
AFLCMC/EZSI 


DISTRIBUTION  A.  Approved  for  public  release:  distribution  unlimited. 


Targeted  Audience  -  Acquisition  center  program 
office,  especially  PMs,  SEs,  and  SSEs 
Training  -TBD 

Accountability  -  Recommended  to  PEOs  as  a  best 
practice 

Sustainment  organization  -  Transition  of  SSE  Acq 
Guidebook  in  FY20  to  AFLCMC/EZSI  and  SMC/ENX 
for  sustainment.  SETR  Toolset  will  be  continue  to  be 
maintained  by  AFLCMC/EZSI. 


Breaking  Barriers  ...  Since  1947 


LOA  2,  Task  1.6  (COMPLETE) 

Provide  recommended  system  security  ianguage  for 

iCDs,  CDDs,  and  CPDs 


Task  Description  &  Deliverables 

Description 

•  Create  guidance  that  enables  program  offices  to 
interact  with  users  and  inform  the  development  of 
weapon  system  requirements  that  account  for  SSE 
activities  throughout  the  acquisition  life  cycle. 


Resource  Loaded  Schedule 


FY16 

FY17 

FY18 

FY19 

FY20 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

SSE  Acq 
Lang  V1 


Deliverables 

•  Updated  SSE  Acquisition  Guidebook  identifying 
process  owners;  summaries  of  applicable 
requirements  development  processes;  and  sample 
ICD,  CDD,  and  CPD  requirements  language 


Any  updates  part  of  Task  1.4 


Imlay,  Martin, 
Zimmerman,  Moyer 


Done  Criteria 


Institutionalization 


•  Approval  of  USAF  SSE  Acquisition  Guidebook 
v1 .1  by  the  CR-TAC 


•  Targeted  Audience  -  Acquisition  center  program  office, 
especially  PMs,  SEs,  and  SSEs 

•  Training  -  See  Task  1 .4 

•  Accountability  -  See  Task  1 .4 

•  Sustainment  organization  -  See  Task  1 .4 


DISTRIBUTION  A.  Approved  for  public  release:  distribution  unlimited. 


Breaking  Barriers  ...  Since  1947 


LOA  2,  Task  1.7 

Develop  system  and  acquisition  security  requirements 

for  programs 


Task  Description  &  Deliverables 

Description 

•  Develop  a  requirements  construct  modeled  after 
the  format  used  in  (MIL-HDBK)  51 6C,  that  focuses 
on  criterion,  standards,  methods  of  compliance  (i.e., 
verification),  and  references. 

Deliverables 

•  Traceable  to  NIST  controls  for  reciprocity  and  audit 
purposes. 

•  Aligned  with  various  domain  frameworks 

•  An  USAF-wide  solution  that  includes  areas  of 
domain-agnostic  requirements 

Done  Criteria 


Resource  Loaded  Schedule 


Zimmerman,  Johnson,  Bumgardner,  Fiore,  Furey- 
Deffely,  Mclnnes,  Aiexander,  Scruggs,  Jones, 
_ Salazar,  Newton _ 

Institutionalization 


•  Approval  of  the  Final  SSE  Requirements  Construct 
by  CR-TAC 

•  Construct  posted  to  site  accessible  by  all 
acquisition  center  program  offices 


•  Targeted  Audience  -  Acquisition  center  program 
office,  especially  PMs,  SEs,  and  SSEs 

•  Training  -  Guidance/instruction  on  use  of  Construct 

•  Accountability  -  Potentially  update  Air  Force 
Instruction  17-101  or  other  instruction 

•  Sustainment  organization  -  Transition  in  FY20  to 
AFLCMC/EZSI  and  SMC/ENX  for  sustainment 


DISTRIBUTION  A.  Approved  for  public  release:  distribution  unlimited. 


Breaking  Barriers  ...  Since  1947 


Objective  2: 
Process  Assessment 


■  Objective  Description:  Develop  metrics  to  measure  SSE 
incorporation  into  SE  processes  and  deliverables 

■  OPR: 

■  Lead:  Mr.  Jeff  Mayer,  AFLCMC/EZC 

■  Representatives  from  AFLCMC,  SMC,  NWC,  DOEs 


DISTRIBUTION  A.  Approved  for  public  release:  distribution  unlimited. 


Breaking  Barriers  ...  Since  1947 
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L0A2,  Task  2.1 

Develop  a  Cyber  Health  Scorecard  to  measure  SSE  process 

health  within  program  offices 


Task  Description  &  Deliverables 

Description 

•  Develop  scorecard  for  program  office  use 

•  Enable  programs  to  evaluate  quality  of  applied 
programmatic  practices 

Deliverables 

•  Final  Enhanced  Guidance 

•  Updated  Overview  and  Training  briefings 

•  Health  Scorecard  Configuration  Management  Plan 

•  Cyber  Health  Scorecard 


Resource  Loaded  Schedule 


Mayer,  Lee,  Hart,  Moyer,  Johnson 


Done  Criteria 


Institutionalization 


•  Final  Cyber/SSE  Health  Assessment 
reviewed/approved  by  the  CR-TAC 

•  Guidance  recommending  use  of  tool  sent  by 
CROWS  or  SAF/AQR  to  PEOs 

•  PEO  Enterprise  Roll-up  Capability  integrated  into 
tool 

•  Assessment  posted  to  site  accessible  by  all 
acquisition  center  program  offices 

DISTRIBUTION  A.  Approved  for  public  release:  distribution  unlimited. 


•  Targeted  Audience  -  Acquisition  center  program 
office,  including  PMs,  System  Program  Directors,  & 
PEOs 

•  Training  -  Narrated  training  briefs  and  enhanced 
guidance 

•  Accountability  -  Memorandum  from  SAF/AQR  to 
PEOs  encouraging  use  of  assessment 

•  Sustainment  organization  -  Potential  transition  in 
FY1 9  to  AF  SE  Assessment  Model  (SEAM)  and 
managed  by  AFMC/ENS  and  SMC/ENE 


Breaking  Barriers  ...  Since  1947 


LOA  2,  Task  2.2 

Develop  methodologies  &  metrics  to  measure  our  systems’ 

security  and  resiliency 


Task  Description  &  Deliverables 


Resource  Loaded  Schedule 


Description 

•  Form  an  AF-level  Cybersecurity  Metrics  Framework 

•  Allows  capturing  and  summing  metrics  to 
provide  system  and/or  platform  level  insight 

•  Conduct  pathfinders,  refine  metrics,  and 
instantiate  a  collection  tool  &  analysis  method 

Deliverables 

•  AF  Cyber  Metrics  Framework 


FY16 

FY17 

FY18 

FY19 

FY20 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

Framework  / 
Metric  Definition 


Pathfinder 


Updates  &  Support 
Materiai  Deveioped 


Mayer,  Lee,  Hart,  Moyer,  Johnson,  SEI 


Done  Criteria 


Institutionalization 


•  Final  AF  Cyber  Metrics  Framework  reviewed/ 
approved  by  the  CR-TAC 

•  Framework  posted  to  site  accessible  by  all 
acquisition  center  program  offices 


•  Targeted  Audience  -  Acquisition  center  program 
office,  including  PMs,  System  Program  Directors,  & 
PEOs 

•  Training  -TBD 

•  Accountability  -  TBD 

•  Sustainment  organization  -  TBD 


DISTRIBUTION  A.  Approved  for  public  release:  distribution  unlimited. 


Breaking  Barriers  ...  Since  1947 


U.S.  AIR  FORCE 


Objective  3: 
Product  V&  V 


■  Objective  Description:  Develop  system  cyber  test  and  evaluation 
methodology  and  capability  across  the  lifecycle  for  all  AF  systems  - 
aircraft,  weapons,  C4ISR,  IT,  space,  nuclear 

■  OPR: 

■  Dr.  Joe  Nichols,  AFTC/CZ 

■  Reps  from  AF/TE,  AFOTEC,  AFMC,  AFLCMC,  SMC,  NWC,  AFRL, 
NASIC,  DOES 


DISTRIBUTION  A.  Approved  for  public  release:  distribution  unlimited. 


LOA2,Task3.1  (COMPLETE) 

Monitor  &  provide  Cyber  T&E  Study 


Task  Description  &  Deliverables 

Description 

•  Complete  Cybersecurity  Test  and  Evaluation  (CTE) 
Study  under  guidance  of  46^^  Test  Squadron 

•  Identify  environment,  infrastructure,  tools, 
methodology,  manpower,  &  resources  required 

Deliverables 

•  Cyber  T&E  Study 

•  Capability  and  infrastructure  gaps 

•  Process  recommendations  &  investment  map 

•  Manpower  study  on  required  expertise  and 
workforce  requirement 


Resource  Loaded  Schedule 


FY16 

FY17 

FY18 

FY19 

FY20 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

T&E 

Study 


Greene 


Done  Criteria 


Institutionalization 


•  Completion  of  the  Cyber  T&E  Study  to  inform 
investment  planning  and  task  3.2 


•  The  Cyber  T&E  Study  is  complete  and  maintained  by 
the  46  TS.  Analysis  will  be  used  to  inform  investment 
planning  and  task  3.2 


DISTRIBUTION  A.  Approved  for  public  release:  distribution  unlimited. 


Breaking  Barriers  ...  Since  1947 


LOA  2,  Task  3.2 

Cyber  Test  Technique  Development 


U.S.  AIR  FORCE 


Task  Description  &  Deliverables 


Resource  Loaded  Schedule 


Description 

•  Develop  cybersecurity  test  strategies,  document 
best  practices  and  lessons  learned,  and  produce  a 
cybersecurity  test  techniques  handbook 


FY16 

FY17 

FY18 

FY19 

FY20 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

w 


CSRA  Guidebook 


Deliverables 

•  Cyber  System  Risk  Assessment  Guidebook 

•  Cyber  T&E  Guidebook 


Done  Criteria 


Hobin,  Newton,  Jones,  Borror,  Camp 


Institutionalization 


•  All  guidebooks  and  methodology  approved  for  use 
by  Headquarters  AF/T&E 

•  46  TS  coordination  and  comment  resolution 
completed 

•  Internal  LOA  2  coordination  and  comment 
resolution  completed 

•  Cross-LOA  coordination  and  comment 
resolution  completed 


•  Targeted  Audience  -  Acquisition  center  program  office 
and  Air  Force  Test  Center,  especially  46  TS 

•  Training  -TBD 

•  Accountability  -  TBD 

•  Sustainment  organization  -  Transition  upon 
completion  to  the  46  TS  for  sustainment 


DISTRIBUTION  A.  Approved  for  public  release:  distribution  unlimited. 


Breaking  Barriers  ...  Since  1947 


Schedule 


Breaking  Barriers  ...  Since  1947 
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Digitally  Trahsfb^l^ir  g  Amspace  &  Defense 


:kheli!f 


Tim  Walden 
Lockheed  Martin  Fellow 
Chief  Enginejer,  Corporate  Engineering 
&  Produchoh  Opecafions 


®  2017  Lockhew  Martin  Corporation.  All  Rights  Reserved 


“Change  is  the  law 
of  life.  And  those 
who  look  only  to  the 
past  or  present  are 
certain  to  miss  the 
future.” 

JOHN  F.  KENNEDY 


©  201 7  Lockheed  Martin  Corporation.  Aii  Rights  Reserved. 


Reshaping  Industrial  Operations 


Machines  evolving  from  tools  to  trusted  human  peers 


©  201 7  Lockheed  Martin  Corporation.  Aii  Rights  Reserved. 
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Government  on  the  Same  Journey 


urta^MiMO 

What  is  a  Digital  Twin? 


“An  integrated  multi  physics,  multiscale,  probabilistic 

simufation  of  an  as-built  system, 

enabled  by  Digital  Thread, 


that  uses  the  best  available  models,  sensor  information,  and 
input  data  to  mirror  and  predict  activities^ performance  over 
the  life  of  its  corresponding  physical  twin.” 

l^^gou^ce,•  DAU  Qiosssr/  Qf  Defense  and  Terms} _ 


A  Digital  Twin  is  NOT: 

•  a  Digital  Tool  for  Configuration  Management 

•  a  3D  Geometric  Model  of  an  As-Built  System 

•  a  Model-based  Definition  of  an  As-Built  System 


©  201 7  Lockheed  Martin  Corporation.  All  Rights  Reserved. 


4 


A  Call  to  Arms 


*  We  must  transition  to  a  structured,  digitai  reiationship,  with  haste 

-  The  integrated  digital  models  must  be  the  unambiguous  source  of  truth 

-  Models,  and  their  data  and  simuiations,  live  forever 

-  Provide  the  necessary  context  for  future  use 

*  We  must  prepare  for  machines  as  human  force-muitipiiers 

-  Structure  the  data 

-  Develop  the  trust 

*  OEMs  and  Government  must  find  a  way  to  experiment  together 

-  Outside  of  the  competition  constraints 

*  We  must  embrace  muitipie  pipeiines  for  skiiis 

-  Explore  avenues  beyond  the  4-year  degrees  for  acquiring  new  skills 

-  Exploit  technology  for  rapid  skill  development  and  training 


©  201 7  Lockheed  Martin  Corporation.  Aii  Rights  Reserved. 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Tutorial  Learning  Objectives 

Learning  Objectives:  At  the  conclusion  of  this  tutorial,  students  should  be  able 

to: 

•  Define  and  distinguish  key  modeling  and  simulation  (M&S)  terms 

•  Name  some  ways  in  which  M&S  can  aid  in  needs  and  opportunities  analysis 

•  Illustrate  the  contents  of  the  five  major  components  of  a  system  effectiveness 
simulation  for  a  system 

•  Explain  typical  applications  of  simulations  in  several  engineering  disciplines 

•  Identify  issues  that  need  to  be  addressed  in  planning  for  M&S  use  during  test 
and  evaluation 

•  Name  some  types  of  models  and  simulations  used  in  the  planning  / 
execution  of  system  production 

•  Explain  how  system  operation  simulations  can  be  used  to  investigate  system 
anomalies  during  sustainment 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Tutorial  Outline 


•  Part  1 :  Overview  of  Modeling  and  Simulation 

•  Part  2:  Use  of  M&S  by  Phase  of  the  Systems  Engineering  Process 

-  M&S  in  System  Needs  and  Opportunities  Analysis 

-  M&S  in  Concept  Exploration  and  Evaluation 

-  M&S  in  Design  and  Development 

-  M&S  in  Integration  and  Test  &  Evaluation 

-  M&S  in  Production  and  Sustainmertt^JP 
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Part  1 : 
Overview  of 

Modeling  and  Simulation 


Coolahan  Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Associates,  LLC 


Lecture  Outline 


Definitions  and  Distinguishing  Characteristics 
Views  and  Categories  of  Models  and  Simulations 
Resolution,  Aggregation,  and  Fidelity 
Overview  of  the  Model/Simulation  Development  Process 
Important  M&S-Related  Processes 
M&S  as  a  Professional  Discipline 
Summary 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Key  Modeling  and  Simulation  Definitions 


There  are  a  number  of  definitions  of  models,  simulations,  and  modeling  and 
simulation  (M&S).  For  the  purposes  of  this  tutorial,  we  will  adopt  the  definitions 
published  by  the  U.S.  Department  of  Defense  (DoD),  below. 

•  Model:  A  physical,  mathematical,  or  otherwise  logical  representation  of  a 
system,  entity,  phenomenon,  or  process.  [1] 

•  Simulation:  A  method  for  implementing  a  model  over  time.  [1] 

•  Modeling  and  simulation:  The  discipline  that  comprises  the  development 
and/or  use  of  models  and  simulations.  [2] 


Sources:  (1 )  Department  of  Defense  Modeling  and  Simulation  (M&S)  Glossary,  July  1 , 201 3; 
available  at  httD://www.msco.mil/MSGIossarv.html 

(2)  DoD  5000.59,  DoD  Modeling  and  Simulation  (M&S)  Management,  August  2007 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Distinguishing  Between  Modeis,  Simulations, 

and  M&S-Reiated  Toois 


•  Models 

-  Need  not  be  computer-based 

-  Represent  something  in  the  real  world 

-  Are  “static”  representations 

•  Simulations 

-  Need  not  be  computer-based  _ 

-  Represent  something  in  the  rea]* *w6rld 

-  Are  “dynamic”  representations  (of  models) 

•  M&S-Related  Tools 

-  Are  typically  computer-based 

-  Do  not,  by  themselves,  represent  something  in  the  real  world 

-  Can  be  used  to  create  (computer-based)  models  and  simulations 

•  Examples 

-  Microsoft  Excel  is  a  “tool”  (not  a  model),  but  can  be  used  to  create  a 
“cost  model”  of  a  system 

-  AnyLogic  is  a  modeling  tool  that  can  be  used  to  create  a  “process 
simulation” 


7 


IM 

Coolahan 
Associates,  LLC 


Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Five  Different  “Views”  of 
Models  and  Simulations 
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Coolahan  Modeling  and  Simulation  in  the  Systems  Engineering  Process 

30|ected  Major  Modeling  &  Simulation 

Application  Domains 


•  Military  systems 

-  Air  and  missile  defense 

-  Strike  warfare 

-  Undersea  warfare 

•  Civilian  systems 

-  Aerospace 

-  Automotive 

-  Electronics 


Homeland  security 

-  Airborne  hazard  dispersion 

-  Disease  spread 

-  Traffic  evacuation 
Medicine 

-  Drug  discovery 

-  Health  care 

-  Surgery  simulation 


Coolahan 
Associates,  LLC 


Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Selected  Major  Modeling  &  Simulation  Roles 


Planning  and  analysis 

-  “How  many  of  system  X  do  I  need?”  “Which  alternative  is  best?” 
Experimentation 

-  “How  could  we  use  this  better?”  “What  might  happen  if  we  tried  this?” 
Systems  engineering  and  acquisition 

-  Principal  focus  of  this  course 
Test  and  evaluation  (T&E) 

-  “Does  the  system  work  as  expected?”  “Will  it  help  in  the  real  world?” 
Training 

-  “How  can  we  ensure  the  system  is  used  correctly?”  “How  can  we  prepare 
pilots  for  rare  emergency  situations?” 

Cost  estimation 

-  “How  much  will  this  cost?”  “How  can  we  reduce  cost?” 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Modeling  and  Simulation  Impiementation 

Techniques 


•  Technique  decisions  to  be  made,  based  on  application 

-  Static  vs.  dynamic 

-  Deterministic  vs.  stochastic  (“Monte  Carlo”) 

-  Discrete  vs.  continuous 

-  Discrete-event  vs.  time-stepped 

-  Standalone  vs.  embedded  (“inHhe  loop”) 

-  Unitary  vs.  distributed 

-  Live  vs.  virtual  vs.  constructive  (more  to  follow  on  next  slide) 

•  Other  technique  decisions 

-  Visualization  needs 

-  Stimulation  of  real  systems 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Categorizing  Simulations  by  the  Nature  of 
Human-System  Interaction 


•  L/Ve  simulation:  A  simulation  involving  real 
people  operating  real  systems 


-  Examples:  exercises,  operational  tests 

•  Virtual  simulation:  A  simulation 
invoiving  real  people  operating 
simuiated  systems 

-  Examples:  cockpit  simulator,  driving 
simulator 

•  Constructive  simulation:  A  simulation 
involving  simulated  people  (or  no 
people)  operating  simulated  systems 


People 

Real  Simulated 
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-  Examples:  crash  test  facilities,  missile 
6-degree-of-freedom  simulations 


Question:  What  would  you  call  a  simulation  involving  simulated  people 
operating  real  systems?  If  the  system  were  an  airplane,  would  you  fly  on  it? 
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Coolahan  Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Categorizing  Models  and  Simulations 
by  Levels  of  Resolution 

Most  M&S  application  domains  have  a  hierarchical  means  of  categorizing 
models  and  simulations  in  that  domain,  by  resolution  level. 


Military  Simulation  Pyramid 

PATRIOT-centric  example 


Gulf  War 

Air  defense 

Missile 

intercept 

Terminal 
guidance 


Campaign 


Mission 


Engagement 


Engineering 


Human  Body  M&S  Pyramid 

Cardiac-centric  example 


More^aggregation 
Shorter  run  time 


Less  aggregation 
Longer  run  time 


Whole  body 

Cardio¬ 

vascular 
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Relative  Run-times  of 


Live,  Virtual,  and  Constructive  Simulations 


Faster  than  real  time 


Real  time 


Siower  than  real  time 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Resolution,  Aggregation,  and  Fidelity 

•  Resolution:  The  degree  of  detail  and  precision  used  in  the  representation  of 
real  world  aspects  in  a  model  or  simulation 

-  Models  and  simulations  at  lower  levels  of  M&S  “pyramid”  tend  to  exhibit 
more  resolution;  this  does  not  necessarily  imply  more  accuracy 

•  Aggregation:  The  ability  to  group  entities  while  preserving  the  effects  of  entity 
behavior  and  interaction  while  grouped 

-  “Campaign-level”  simulations  often  aggregate  military  entities  into  larger 
groups  (e.g.,  brigades  vs.  battalions) 

•  Fidelity:  The  accuracy  of  the  representation  when  compared  to  the  real 
world 

-  Greater  fidelity  does  not  imply  greater  resolution 


Source  of  definitions:  Department  of  Defense  Modeling  and  Simulation  (M&S)  Glossary,  July  1 , 

2013;  available  at  httD://www.msco.mil/MSGIossarv.html 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


The  Model/Simulation  Development  Process 


•  Developing  a  model  or  simulation  is,  in  itself,  a  type  of  “systems  engineering” 
process 

•  Although  shown  below  as  a  “waterfall,”  various  forms  of  iteration  are 
possible. 


Requirements 

definition 


Conceptual 

analysis 


Sequence 


Design  and 
development 


Iteration 


Integration 
and  testing 


Execution  and 
evaluation 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Important  M&S-Related  Processes: 
Configuration  Management 


•  Configuration  management  is  just  as  important  for  M&S  as  it  is  for  systems 
and  software  engineering. 

•  Issues  in  model  /  simulation  configuration  management 

-  Identifying  the  “current  version”  during  development 

-  Maintaining  a  copy  of  each  “release” 

-  Tracking  defects  and  their  correction 

-  Maintaining  records  of  recipients  of  each  version 

-  Managing  multiple  “branches”  for  multiple  users 

-  Managing  co-developed  versions  if  source  is  distributed 

-  Incorporating  externally-made  changes  in  a  “baseline”  version 

-  Regression  testing  of  new  versions 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Important  M&S-Related  Processes: 
Verification,  Validation,  and  Accreditation  (VV&A) 


•  Verification  -  The  process  of  determining  that  a  model  or  simulation 
implementation  and  its  associated  data  accurately  represent  the  developer's 
conceptual  description  and  specifications 

-  Did  we  build  the  model  right? 

•  Validation  -  The  process  of  determining  the  degree  to  which  a  model  or 
simulation  and  its  associated  data  are  an  accurate  representation  of  the  real 
world  from  the  perspective  of  the  intended  uses  of  the  model 

-  Did  we  build  the  right  model? 

•  Accreditation  -  The  official  certification  that  a  model  or  simulation  and  its 
associated  data  are  acceptable  for  use  for  a  specific  purpose 

-  Is  this  the  right  model  to  use  for  this  purpose? 


Source:  DoD  Instruction  (DoDI)  5000.61  DoD  Modeling  and  Simulation  (M&S) 
Verification,  Validation,  and  Accreditation  (VV&A),  Decembers,  2009 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Interoperable  Simulation: 

The  High  Level  Architecture  (HLA) 


•  Architecture  calls  for  a 
federation  of 
simulations 

•  Architecture  specifies 

-  Ten  Rules  that  define 
relationships  among 
federation  components 

-  An  Object  Model 
Template  that  specifies 
the  form  in  which 
simulation  elements  are 
described 

-  An  Interface 
Specification  that 
describes  the  way 
simulations  interact 
during  operation 


C++ 

Ada  95 
CORBA  IDL 


Java 

Runtime  Infrastructure  (RTI) 

Federation  Management  Declaration  Management 

Object  Management  Ownership  Management 

Time  Management  Data  Distribution  Management 


The  HLA  was  originally  developed  by  DoD.  It  is  now  IEEE  standard  1516. 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Modeling  and  Simulation 
as  an  Academic  Discipline 


•  Very  few  Universities  offer  Modeling  &  Simulation  as  an 
academic  discipline  with  a  degree  program 

•  Graduate-level  M&S  degree  programs  are  offered  in  the 
U.S.  by: 

-  The  University  of  Centrai  Fiorida  (UCF) 

-  Oid  Dominion  University  (dtJU) 

-  The  University  of  Aiabama  in  Huntsviiie  (UAH) 

-  The  Naval  Postgraduate  School  (NPS) 

-  Arizona  State  University  (ASU) 

-  Purdue  University  Calumet 

-  Philadelphia  University 

•  M.S.  degree  concentrations  in  M&S  are  offered  by: 

-  The  Johns  Hopkins  University  (JHU)  [in  Systems  Engineering] 

-  Coiumbus  State  University  (GA)  [in  Applied  Computer  Science] 
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Modeling  and  Simulation 
as  a  Professional  Discipline 


•  Professional  certification  in  M&S  is  available 

-  Certified  Modeiing  and  Simulation  Professional  (CMSP) 
designation 

■  Originated  by  the  National  Training  and  Simulation 
Association  (NTSA) 

■  Now  administered  by  the  Modeling  and  Simulation 
Professional  Certification  Commission  (M&SPCC) 

-  Requirements: 

■  Relevant  (simulation)  work  experience  and  educational 
requirements,  three  letters  of  recommendation,  and  a  passing 
grade  on  the  exam 

■  Fee  of  $250 

■  14  days  allowed  to  answer  100-question  examination 

-  See  web  site:  http://www.simprofessional.ora 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Module  Summary 


•  A  model  is  a  physical,  mathematical,  or  otherwise  logical  representation  of  a 
system,  entity,  phenomenon,  or  process.  A  simulation  is  a  method  for 
implementing  a  model  over  time. 

•  Models  and  simulations  can  be  categorized  by  their  application  domain,  role, 
implementation  techniques,  method  of  human  interaction,  and  level  of 
resolution. 

•  Developing  a  model  or  simulation  isrih  itself,  a  type  of  systems  engineering 
process. 

•  Configuration  management  and  VV&A  are  two  important  M&S  processes. 

•  Simulations  may  be  made  to  interoperate  with  one  another  using  various 
techniques,  including  the  HLA  (IEEE  1516). 

•  M&S  has  not  completely  emerged  as  a  separate  academic  discipline,  but  is 
beginning  to  be  recognized  as  a  professional  discipline. 
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The  “V”  Model  of  Systems  Engineering 


Define  System 
Requirements 


Allocate  System 
Functions  to 
Subsystems 


Testing 


Detail  Design 
of  Components 


Full  System 
Operation  and 
Verification 


Verification  of 
Subsystems 


Verify 

Components 


25 


IM 


Coolahan 
Associates,  LLC 


Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Defense  Acquisition  Program  Model 

(for  Hardware- 1  intensive  Program) 


Development 

RFP 

CDD  Release 

Validation  Decision 


Materiel 

Development 

Decision 


Full  Rate 
Production 
(FRP)  Decision 


Initial  Full 

Operational  Operational 
Capability  Capability 
(IOC)  (FOC) 


0  A\  ^ ^ 


LRIP 

Operational  Test  and 
Evaluation  (OT&E) 

\  1 

Sustainment  DisposajJ 

Materiel 

Solution 

Anaiysis 

Technoiogy 
Maturation  & 
Risk 

Reduction 

Engineering  & 
Manufacturing 
Deveiopment 

Production  & 
Depioyment 

Operations  &  Support 

Legend:  /\  =  Milestone  Decision  ^  =  Decision  Point 


Five  other  variants  of  this  program  motdel  exist  for  other  types  of  programs. 


Source:  DoD  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System,  January  7,  2015 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


A  Representative  Six-Stage  System  Life  Cycle 


Concept 

Development 

Production 

Utilization 

Support 

Retirement 

stage 

stage 

stage 

stage 

stage 

stage 

Source:  ISO/IEC  TR  1 9760,  Systems  engineering  —  A  guide  for  the  appiication  of  ISO/iEC  15288 
(System  iife  cycie  processes),  2003 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

A  Textbook  Representation  of  Systems 
Engineering  Stages  &  Phases 


Systems 

Engineering 

Stages 

Concept  Development 

Engineering  Development 

_ 

Post  Development 

Systems 

Engineering 

Phases 

Needs 

Analysis 

Concept 

Exploration 

Concept 

Definition 

Advanced 

Development 

Engineering 

Design 

Integration  & 
Evaluation 

Production 

Operations 
&  Support 

Source:  Systems  Engineering:  Principies  and  Practice,  Kossiakoff,  A.,  Sweet,  W.  N.,  Seymour,  S.  J., 
and  Biemer,  S.  M.,  Wiley,  2011. 


28 


Coolahan 
Associates,  LLC 


Modeling  and  Simulation  in  the  Systems  Engineering  Process 

A  Reference  Model  of  the 
Systems  Engineering  Process  for  this  Tutorial 


•  System  Needs  and  Opportunities  Anaiysis 

-  Defining  and  validating  needs,  and  determining  feasibility 

•  Concept  Expioration  and  Evaiuation 

-  Exploring  and  evaluating  system  concepts,  refining  required  performance 
characteristics  and  required  effectiveness  in  representative  operational 
environments,  and  performing  analysis  of  alternative  concepts 

•  Design  and  Deveiopment  Wp 

-  Designing  and  prototyping  the  system,  providing  for  human-system 
integration,  refining  performance  estimates,  and  production  planning 

•  Integration  and  Test  &  Evaluation  (T&E) 

-  Integrating  the  system  components,  and  testing/evaluating  the  system  in 
representative  environments 

•  Production  and  Sustainment 

-  Producing  and  sustaining  the  system,  including  providing  for  reliability, 
availability,  logistics,  and  training 
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Comparison  of  System  Life  Cycle  Models 


DOD 

5000.02 

(Hardware- 

Intensive 

Systems), 

2015 

Materiel 

Solution 

Analysis 

Technology  Maturation 
&  Risk  Reduction 
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Production 

& 

Deployment 

Operations  &  Support 
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15288, 

2003 

Concept 

Development 

Production 

Utiliza¬ 
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Support 

Retire¬ 
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Kossiakoff 

Textbook 

(Stages), 

2011 

Concept  Development 

Engineering  Development 

Post- Development 

Kossiakoff 

Textbook 

(Phases), 

2011 

Needs 

Analysis 

Concept 

Exploration 

Concept 

Definition 

Advanced 

Develop¬ 

ment 

Engineering 

Design 

Integration 

& 

Evaluation 

Production 

Operations  &  Support 

This 

Course 

System 
Needs  & 
Opportu¬ 
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Analysis 

Concept  Exploration  & 
Evaluation 

Design  &  Development 

Integration 
and  Test  & 
Evaluation 

Production  &  Sustainment 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Module  Objectives  and  Outline 


Module  Objective: 

•  To  describe  the  use  of  modeling  and  simulation  in  the  system  needs  and  opportunities  analysis 
phase  of  the  systems  engineering  process. 

Module  Outline 

•  Needs  vs.  Opportunities  for  New  or  Improved  Systems 

•  The  U.S.  Military  Process  for  Capabilities-Based  Assessment 

•  Commercial  System  Processes 

•  M&S  Use  in  Operational  Analysis 

•  M&S  Use  in  Functional  Analysis 

•  M&S  Use  in  Feasibility  Determination 

•  Summary 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Needs  vs.  Opportunities  for  New  or  Improved  Systems 

•  New  or  improved  systems  can  be  initiated 

-  As  the  result  of  the  need  for  a  new  or  improved  capability;  or 

-  To  take  advantage  of  an  opportunity 

•  For  military  systems 

-  A  need  can  result  from  the  emergence  of  a  new  threat 

-  An  opportunity  can  arise  because  of  a  technology  breakthrough 

•  For  commercial  systems  ■ 

-  A  need  can  result  from  a  new  legal  or  regulatory  requirement 

-  An  opportunity  can  arise  from  a  new  demand  in  the  marketplace  or 
financial  incentives  to  provide  an  improved  capability  (e.g.,  hybrid  autos) 

•  M&S  can  be  used  to 

-  Explore  the  effectiveness  or  utility  of  a  new  concept 

-  Estimate  the  cost  of  envisioned  alternatives 

-  Aid  in  determining  feasibility  of  a  new  or  improved  system 


31 


IM 


Coolahan 
Associates,  LLC 


Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Capabilities-Based  Assessment  (CBA)  Process  in  the  U.S. 
Joint  Capabilities  Integration  and  Development  System 


Source:  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  -  A  Primer 


Cooiahan  Modeling  and  Simulation  in  the  Systems  Engineering  Process 

iociales,  LLC  . 

Capabilities-Based  Assessment 
Needs  Assessment  Task  Flow 


Source:  Capabilities-Based  Assessment  (CBA)  User’s  Guide 
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Commercial  Processes  for  Identifying  and  Analyzing 

Needs  and  Opportunities 


•  Commercial  processes  can  vary  depending  on  the  industry  and  the  individual 
company 


•  In  general,  there  is  a  fairly  continual  operational  analysis  process,  which 
periodically  triggers  a  functional  analysis  based  on  a  set  of  operational 
objectives,  followed  by  a  feasibility  determination  resulting  in  operational 
requirements  for  a  new  or  improved  system 


Technology  Feasibility  Measures  of 

Improvements  Criteria  Effectiveness 


Operational 

Requirements 


Legacy  /  Similar  System  Information 


Simplified  Needs  and  Opportunities  Analysis  Diagram 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Modeling  and  Simulation  Use 
in  Operational  Analysis  (1  of  3) 


•  Simulations  of  (Relative)  Performance 

-  Although  the  absolute  performance  of  a  system  will  generally  not 
decrease  overtime  (and  will  often  increase  through  upgrades),  its 
relative  performance  eventually  degrades 

■  A  new  missile  threat  may  have  capabilities  outside  the 
performance  envelope  of)an  air  defense  system 

■  Competing  products  may  incorporate  new  technology  (e.g., 
cell  phone  decreasing  size  and  weight,  longer  battery  life) 

-  Simulations  of  the  threat  or  competitive  environment  must  be 
continually  executed  to  predict  system  obsolescence 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Modeling  and  Simulation  Use 
in  Operational  Analysis  (2  of  3) 


•  Models  of  Total  Ownership  Cost 

-  Changing  costs  for  operations  and  maintenance  labor  or 
consumables  may  impact  how  much  a  user  must  pay  to  own  the 
product 

■  At  certain  thresholds  of  the  price  of  gasoline,  ownership  of 
vehicles  with  higher  gasoline  consumption  can  become 
unaffordable  ' 

-  Models  of  all  ownership  costs  must  be  developed  and  maintained 

•  Models  of  Sustainability 

-  At  some  point  in  time,  parts  for  a  given  system  implementation 
may  no  longer  be  available,  at  any  cost 

-  Models  of  parts  availability  must  be  developed  and  maintained 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Modeling  and  Simulation  Use 
in  Operational  Analysis  (3  of  3) 


•  Value  Modeling  Tools 

-  Some  value  attributes  of  systems  defy  quantitative  engineering 
measurement 

■  “Intelligence  estimates”  of  the  performance  and  fielding  date  of 
future  threats  are  dependent  on  judgment  of  subject  matter 
experts  (SMEs) 

■  “Stylishness”  of  new  cars  is  in  the  eyes  of  the  beholders 

-  Models  of  value  using  multiple  unrelated  measures  need  to  be 
constructed 

■  Value  attributes  must  be  identified 

■  Measures  for  collecting  valid  opinions  and  quantifying  them 
must  be  devised 

■  A  “weighting  scheme”  must  be  applied  in  the  model 


37 


IM 


Coolahan 
Associates,  LLC 


Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Illustration  of  M&S  Use  in  Operational  Analysis 


•  Simulation  of  System  Operations  through  Games 

-  Can  be  a  structured  “war  game”  with  blue,  red,  white,  green  cells 

-  Can  be  a  “seminar”  game  with  subject  matter  experts  in  various  fields 
working  collaboratively 

-  Can  be  used  to  explore  concepts  of  operations  for  proposed  systems 


-  The  term  “serious  games”  has  come  into  vogue  to  describe  these 


Seminar  Game  Example: 

•  How  would  I  use  the  existing 
system  in  this  scenario? 

•  What  technology  improvements 
could  be  made? 

•  If  I  had  a  system  with  this 
capability,  what  would  I  do  now 
in  this  situation? 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Modeling  and  Simulation  Use 
in  Functional  Analysis 


•  Functional  analysis  needs  to  translate  operational  system  objectives 
into  system  functions 

-  Essentially,  a  feasible  concept  must  be  able  to  be  “envisioned” 

-  In  a  need-driven  process,  some  system  functions  might  be 
relatively  well-known  from  legacy  systems 

•  Deriving  a  functional  structure  contains  elements  of  art  /  architecting 

•  Modeling  tools  can  be  used  to  develop  a  system  functional 
breakdown 

-  Can  start  with  a  relatively  simple  block  diagram  (e.g.,  Microsoft 
Visio  or  PowerPoint  could  be  used  to  generate  a  top-level  “model” 
of  a  system) 

-  More  formal  notations  can  be  used  to  ensure  inputs  and  outputs 
are  properly  considered  (e.g.,  IDEFO  diagrams  or  Unified 
Modeling  Language  (UML)  diagrams) 
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Modeling  and  Simulation  Use 
in  Feasibility  Determination  (1  of  3) 


•  Simulations  of  System  Operational  Effectiveness  -  Input  Needs 

-  Estimates  of  the  performance  of  an  envisioned  system  implementation, 
at  a  less-detailed  level,  such  as 


■  Probability  of  detection  as  a  function  of  target  cross-section  and 
range  (in  various  environments)  for  a  radar  system 

■  Miles  per  gallon  as  a  functionjDf  fuel  octane,  temperature,  and 
pressure  for  an  automobile 

Similar  estimates  for  systems  with  which  the  envisioned  system  must 
interact  collaboratively  or  cooperatively 

For  systems  with  competitive  adversary  systems,  similar  estimates  for 
each  adversary  system 

Representations  of  the  natural  environment  (land,  sea,  and/or  air),  often 
time-varying 


-  A  model  of  one  or  more  representative  scenarios  of  use  of  the  system, 
including  such  things  as  geographic  location,  environmental  conditions, 
time  of  day,  system  behaviors,  etc. 
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Modeling  and  Simulation  Use 
in  Feasibility  Determination  (2  of  3) 


•  Models  of  Total  Ownership  Cost 

-  Similar  to  those  used  during  operational  analysis 

•  Models  of  Sustainability 

-  Reliability  models  (at  a  relatively  high  level,  unless  data  on  similar 
legacy  system  components  are, available) 

-  Availability  models  (percentage  of  time  the  system  will  be  ready 
when  called  upon) 

-  Maintainability  models  (e.g.,  time  to  repair) 

-  Logistics  support  simulations 
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Modeling  and  Simulation  Use 
in  Feasibility  Determination  (3  of  3) 


•  For  systems  that  are  improvements  to  existing  systems  and/or  use 
iegacy  components,  modeis  and  simuiations  of  those  systems  / 
components  can  be  used  as  a  starting  point 

•  The  outputs  of  modeis  and  simulations  in  the  feasibiiity  determination 
phase  are  generaiiy  various  estimated  measures  of  effectiveness  for 
a  particular  envisioned  system  irnpiementation 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Illustration  of  M&S  Use  in  Feasibility  Determination 

•  Campaign-level  Simulations 

-  Use  Measures  of  Performance  (MoPs)  of  systems  as  inputs 

-  Simulate  system  operation  in  a  computer-based  operational  environment 

-  Produce  Measures  of  Effectiveness  (MoEs)  as  outputs 

-  Can  be  used  to  answer  “so  what”  questions  for  proposed  new  systems 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Module  Summary 


•  New  or  improved  systems  can  be  initiated  as  the  result  of  the  need  for  a  new  or 
improved  capability,  or  to  take  advantage  of  an  opportunity 

•  For  both  military  capabilities  and  commercial  systems,  there  are  somewhat  similar 
approaches  to  needs/opportunities  analysis,  but  using  different  terminology 

•  Value  modeling  tools  are  often  useful  during  operational  analysis  to  help  quantify 
SME  opinions 

•  Formal  modeling  notations  and  tools  are  useful  in  adding  rigor  to  system  functional 
breakdowns 

•  Operational  effectiveness  simulations  are  important  in  performing 

-  Ongoing  operational  analysis  to  determine  operational  objectives  for  new  or 
improved  systems 

-  Analysis  of  envisioned  system  implementations  to  determine  feasibility 

•  Cost  models  must  consider  the  total  ownership  cost  of  systems,  not  just  the 
development  cost 

•  Sustainability  (reliability,  availability,  maintainability,  logistics)  models  and  simulations 
are  also  of  significant  importance  in  operational  analysis  and  feasibility  determination 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Module  Objective  and  Outline 


Module  Objective:  To  describe  the  use  of  modeling  and  simulation  in  the  concept 
exploration  and  evaluation  phase  of  the  systems  engineering  process. 

Module  Outline 

•  Scope  of  Concept  Exploration  and  Evaluation 

•  A  Simplified  Process  Model  for  Concept  Exploration  and  Evaluation 

•  Effectiveness  Simulations 

-  Components  of  Effectiveness  Simulations 

•  Analyses  of  Alternatives 

-  System  Effectiveness  Simulation 

-  Cost  Modeling 

•  Ensuring  a  “Level  Playing  Field” 

•  Summary 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Scope  of  Concept  Exploration  and  Evaluation 

(1  of  2) 


•  Concept  Exploration 

-  Involves  translating  the  operational  requirements  for  the  system 
into  engineering-oriented  performance  requirements  for  the 
system 

■  interpret,  but  do  not  replace,  the  operational  requirements 

-  Several  alternative  candidateTystem  concepts  are  envisioned, 
and  their  performance  characteristics  established 

-  Can  sometimes  be  relatively  limited,  to  only  particular  functions  or 
portions  of  a  legacy  system 

-  For  new  systems,  a  more  creative,  non-prescriptive  method  is 
indicated  that  is  akin  to  systems  architecting 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Scope  of  Concept  Exploration  and  Evaluation 

(2  of  2) 


•  Concept  Evaluation 

-  Involves  taking  the  alternative  concepts  produced  during  concept 
exploration,  defining  them  even  further,  and  evaluating  them 

-  May  be  done  by  a  single  organization  or,  in  the  case  of  a  major 
system  development  by  separate  organizations  in  a  competitive 
environment,  with  an  independent  organization  evaluating  those 
concepts 

-  Results  in  a  selected  system  concept  and  a  set  of  system 
functional  specifications  suitable  to  enter  development 


54 


IM 


Coolahan 
Associates,  LLC 


Modeling  and  Simulation  in  the  Systems  Engineering  Process 

A  Simplified  Process  Model  for 
Concept  Exploration  and  Evaluation 


— > 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Use  of  Legacy  /  Similar  System  Information 

and  M&S  Tools 


•  Reuse  of  models/simulations  is  usually  cost-effective 

-  Usually  require  some  adaptation 

-  Need  subject  matter  experts  &  M&S  professionals  familiar  with  tools 

-  Less  experienced  teams  can  make  use  of  M&S  repositories  /  registries  to 
assist  in  discovery  process 

•  Issues  to  be  aware  of 

-  Lack  of  awareness  of  existing  M&S  tools 

-  “Not  invented  here”  (NIH)  syndrome 

-  Force-fit  of  familiar  tools  (“we’ve  always  used  this  one”) 

•  Best  to  do  selection  based  on  objective  set  of  requirements  /  criteria 

•  Availability  of  authoritative  data  can  be  an  issue 

-  Authoritative  data  on  military  threat  systems  may  be  hard  to  obtain 

-  Authoritative  data  on  “friendly”  systems  may  require  time-consuming 
release  approval 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Effectiveness  Simulations 


•  Effectiveness  simulations  -  typically  at  the  “mission  level”  of  the  military 
simulation  pyramid 

•  Generally  use  parameterized  system  performance  data  generated  by 
performance  simulations 

•  Major  components  of  effectiveness  simulations 

-  The  system  representation  (in  peiformance  terms) 

-  The  system’s  concept  of  operations 

-  The  representation  of  threats  and  friendly  systems 

-  The  representation  of  the  natural  and  man-made  environment 

-  The  scenario 

•  Supporting  elements  of  effectiveness  simulations 

-  User  interface 

-  Data  input  mechanisms 

-  Results  output  mechanisms 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Effectiveness  Simulations  - 
Representing  the  System 


•  During  concept  exploration  and  evaluation 

-  Only  early  estimates  of  system  performance  may  be  available 

-  Systems  based  on  legacy  components  typically  have  more  credible 
representations  than  those  based  on  new  technology 

•  Can  sometimes  use  effectiveness  simulations  for  screening 


-  “If  we  could  build  a  system  with  this  performance,  would  it  make  a 


difference?” 


r 


-  Does  achieving  desired  performance  require  unrealistic  operational 
conditions? 


•  System  performance  typically  represented  parametrically 

-  Using  equations 

-  Using  tables  of  two  or  more  dimensions 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Effectiveness  Simulations  - 
Concept  of  Operations 


•  Need  to  represent  how  the  system  is  employed  in  practice 

-  Concept  of  operations  (CONORS)  can  affect  system  performance 

•  Examples  of  CONORS  effects  on  performance 

-  Submarine  towed  array  system  performance  affected  by  dynamic 
movement  during  submarine  maneuvers 

-  Ground-based  system  may  not  be  activated  until  cued  by  a  surveillance 
system 

-  Automotive  system  may  only  be  activated  when  commanded  by  the 
driver 

-  Flight  performance  of  aircraft  affected  by  formation  flying 

■  Why  do  ducks  fly  in  a  V  formation? 
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Effectiveness  Simulations  - 
Threats  and  Friendly  Systems 


Virtually  every  system,  whether  commercial  or  military,  will  need  to  interact 
with  other  systems 

-  Need  to  represent  other  systems  to  the  degree  that  their  performance 
could  impact  the  system’s  effectiveness 

-  For  commercial  systems,  most  are  cooperative,  or  at  least  neutral 

-  For  military  systems,  must  take  into  account  the  performance  of: 

■  Threat  (“red”)  systems 

■  Cooperating  (friendly,  or  “blue”)  systems 

■  Neutral  (“green”)  systems  (important  in  “irregular  warfare”) 

Level  of  detail  at  which  such  systems  need  to  be  represented  depends  on 
nature  of  potential  interactions  with  system  being  studied 


-  If  neutral  systems  are  only  “clutter,”  can  be  modeled  simply 

-  Some  cooperative  systems  may  only  need  to  be  modeled  as  a  source  of 
communication  messages,  with  a  probability  of  successful  delivery 


-  But  some  threat  systems  need  detail  commensurate  with  system  being 
studied  (e.g.,  threat  aircraft  in  a  “dogfight”  scenario) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Effectiveness  Simulations  - 
The  Natural  and  Man-Made  Environment  (1  of  2) 


•  The  effectiveness  of  virtually  all  systems  is  dependent  on  the  effects  of  the 
natural  and  the  man-made  environments  that  it  encounters  during  operation 

-  Some  effects  are  well  known  and  tolerable  (e.g.,  automobile  radio  or 
Global  Positioning  System  (GPS)  reception  in  middle  of  two-mile  tunnel; 
cell  phone  performance  in  urban  canyons) 

-  Some  effects  are  not  tolerable  (e.g.,  automobile  engine  overheating  in 

Death  Valley)  ^ 

•  Environmental  conditions  are  typically  more  important  for  military  (and  law 
enforcement,  and  other  government)  systems,  which  are  needed  to  operate 
with  high  reliability  in  more  stressful  environments  than  commercial  systems 

-  Dust  storms  for  ground  vehicles,  jamming  environments  for 
communication  systems,  and  supersonic  airflows  for  aircraft 

-  In  other  cases,  some  degradation  of  performance  can  be  tolerated,  but 
needs  to  be  quantified  (e.g.,  sonar  performance) 

•  Effectiveness  simulations  must  model  environmental  conditions  with  fidelity 
commensurate  with  their  effects  on  the  system. 


61 


Coolahan 
Associates,  LLC 


Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Effectiveness  Simulations  - 
The  Natural  and  Man-Made  Environment  (2  of  2) 


•  Models  of  the  natural  environment  include 

-  Atmospheric  characteristics,  such  as  temperature,  pressure,  humidity, 
and  wind  speed  -  for  airborne  systems  and  electromagnetic  propagation 

-  Ground  terrain  characteristics,  such  as  height  vs.  position  and  soil 
properties  -  for  ground-based  systems  and  line-of-sight  calculations 

-  Ocean  characteristics,  such  as  djpth,  sound  velocity  profile,  and  wave 
height  -  for  maritime  systems 

-  Space  characteristics,  such  as  solar  flares  and  sun  spots  -  for  satellite 
reliability/availability  and  electromagnetic  propagation. 

•  Models  of  the  man-made  environment  include 

-  Building  sizes  and  shapes  -  for  line-of-sight  calculations,  and  urban  wind 
velocity  /  contaminant  propagation 

-  Road  networks  -  for  transportation  modeling 

-  Electromagnetic  emissions  -  for  electromagnetic  interference 
calculations 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Effectiveness  Simulations  - 
Scenarios 


•  Scenarios  often  start  out  as  high-level  text  descriptions 

-  But  must  be  quantified  to  be  used  in  effectiveness  simulations 

•  Scenarios  for  a  military  simulation  will  typically  include 

-  The  numbers  and  types  of  each  friendly,  threat,  and  neutral  system 
involved 

-  System  concepts  of  operation,  and  the  way  in  which  entities  move  (either 
scripted,  or  in  some  reactive  way) 

-  Location  and  extent  of  the  “play  box(es)” 

-  Instantiations  of  the  natural  and/or  man-made  environment,  sometimes  in 
great  detail  (e.g.,  Digital  Terrain  Elevation  Data  (DTED)  terrain  files) 

-  A  time  of  year  (important  for  choosing  appropriate  atmospheric  and 
maritime  data) 

-  A  duration,  which  could  range  from  as  little  as  seconds  for  a  missile 
intercept  to  days  or  weeks  for  an  extended  ground  battle 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Analyses  of  Alternatives 


•  An  Analysis  of  Alternatives  (AoA)  is  an  analytical  comparison  of  the 
operational  effectiveness,  suitability,  and  life-cycle  cost  (or  total  ownership 
cost,  if  applicable)  of  alternatives  that  satisfy  established  capability  needs. 

•  Involves  performing 

-  Selection  of  alternatives 

-  Determination  of  effectiveness  measures 


Effectiveness  analysis 
Cost  analysis 

Cost-effectiveness  comparisons 


AoA 
Siuay  -» 
Phn 


Relationship  of  AoAs  to  the  Defense  Acquisition  Process 

Source:  Defense  Acquisition  Guidebook 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Analyses  of  Alternatives  - 
System  Effectiveness  Simulation 


Measures  of 
Effectiveness 
(MOEs) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Analyses  of  Alternatives  - 
Cost  Modeling 


Need  to  consider  all  elements  of  system  cost: 

-  Development  cost 

-  Production  cost 

-  Support  (repairs,  logistics,  training,  etc.)  cost 

-  Disposal  cost 
Development  cost  modeling 

-  Need  to  assess  development  risk;  cost  uncertainty 
Production  cost  modeling 

-  Need  to  account  for  manufacturing  systems  development  cost,  number 
of  units 

Support  cost  modeling 

-  Support  cost  is  usually  the  largest  element  of  total  cost  (-50%) 

-  Need  to  consider  life  of  system,  number  of  operators,  logistics  system 
Disposal  cost  modeling 

-  Often  neglected;  need  to  consider  hazardous  materials 


67 


IM 


Coolahan 
Associates,  LLC 


Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Analyses  of  Alternatives  - 
Ensuring  a  “Level  Playing  Field” 


•  When  comparing  system  alternatives,  need  to  ensure  that  each  system  is 
modeled  “fairly”  with  respect  to  other  systems 

•  Need  to  model  systems  themselves  at  similar  levels  of  resolution 

•  Need  to  take  into  account  key  concepts  of  operation  for  each  system 

-  For  example,  energy  management  for  some  radar  systems 

•  Need  to  model  aspects  of  environrnent  at  appropriate  levels  of  detail 

-  For  example,  line  of  sight  for  ground-based  weapon  systems 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Module  Summary 


•  Concept  exploration  and  evaluation  devises  and  evaluates  a  number  of 
alternative  system  concepts 

•  Reuse  of  existing  models  and  system  effectiveness  simulation  tools,  and  of 
data  on  legacy  systems,  can  often  be  useful  in  this  phase 

•  Effectiveness  simulations  include 


-  The  system  representation  (in  pe#ormance  terms) 

-if(r 

-  The  system’s  concept  of  operations 


-  Representations  of  threats,  friendly  systems,  and  the  natural  and  man¬ 
made  environment 


-  Scenarios  of  system  use 

•  An  analysis  of  alternatives  (AoA)  is  typically  performed  for  major  defense 
systems,  and  employs  both  system  effectiveness  simulations  and  cost 
models 


•  When  used  to  compare  the  effectiveness  of  alternative  systems,  simulations 
must  ensure  a  “level  playing  field”  for  all  of  the  systems 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Scope  of  Design  and  Development 

•  The  design  and  development  phase  of  systems  engineering,  as  discussed  in 
this  course,  refers  to  the  combination  of  the  following  in  Systems 
Engineering:  Principles  and  Practice  [1]: 

-  Advanced  Development 

-  Engineering  Design 

•  Design  and  development  takes  a  system  concept  as  input,  and  transforms  it 
into  a  set  of  realized  system  components  that  are  ready  for  system 
integration  and  testing 


Source:  (1)  Kossiakoff,  A.,  Sweet,  W.  N.,  Seymour,  S.  J.,  and  Biemer,  S.  N.,  Systems 

Engineering:  Principles  and  Practice,  Second  Edition,  John  Wiley  &  Sons,  Inc., 
Hoboken,  N.  J.  (2011). 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


A  Simplified  Process  Model  for 
Design  and  Development 


Defined  , _ ,  Validated 


—  System  — > 

— Development— > 

Concept 

Advanced 

Model 

Engineering 

System 

Development 

System 

Design 

■  Functional  i> 

—  Design  -o 

Specifications 

Specifications 

_Engineered _ ^ 

Prototype 

Test 

Requirements 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Distinguishing  Characteristics  of  M&S  Use  in 
Design  and  Development 


•  Most  of  the  simulations  used  during  design  and  development  fall  within  the 
“engineering”  level  of  the  four-level  (military)  simulation  pyramid 

-  They  usually  model  individual  components  of  the  system 

-  They  often  execute  slower  (or  much  slower)  than  real  time 

-  In  many  cases,  they  need  to  interface  with  one  another  to  represent  a 
subsystem  or  the  system  as  a  whole 

-  They  produce  data  useful  as  input  for  engagement-level  simulations 

•  Whereas  the  earlier  phases  of  the  systems  engineering  process  may  utilize  a 
relatively  small  number  of  models  and  simulations,  in  Design  and 
Development,  there  is  typically  a  large  number  of  rather  diverse  models  and 
simulations  that  are  employed. 

•  Just  as  a  systems  engineer  typically  needs  broad  expertise  to  “ask  the  right 
questions”  across  a  range  of  engineering  disciplines  during  Design  and 
Development,  a  systems  engineer  responsible  for  M&S  needs  to  have  a 
broad  view  of  M&S  tools  that  can  be  applied  in  a  range  of  disciplines  during 
this  phase. 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Range  of  Engineering  Disciplines  Needed  for  System 
Design  and  Development  Simulations 


• 

structural  mechanics/dynamics 

•  Acoustic  propagation 

• 

Fluid  dynamics 

•  Electromagnetic  propagation 

• 

Thermal  analysis 

•  Optical  systems  engineering 

• 

Propulsion 

•  Software  engineering 

• 

Materials  engineering 

•  Manufacturing  process  modeling 

• 

Printed  circuit  design 

•  Cyber  security 

• 

Electrical  power  system  design 

•  T  raff  ic  flow 

• 

Guidance,  navigation  and  control 

•  Human-systems  integration 

• 

Communication  systems  engineering 

•  Crowd  dynamics 

• 

Computer  network  engineering 

•  Human  behavior 

Example  M&S  tools  for  many  of  these  areas  are  cited  in  the  next 
section.  Mention  of  a  specific  M&S  tool  does  not  imply  endorsement. 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Structural  Mechanics/Dynamics  Simulations 


•  Typical  Applications: 

-  Finite  element  analysis 

-  Dynamic  load  analysis 

•  Examples: 

-  NASTRAN  (originally  from  “NASA  Structural  Analysis”  in  the  late  1960s) 


LS-DYNA®  (Livermore  Software  Technology  Corp.) 


MSC  Nastran  result  (source:  Wikipedia) 


LS-DYNA  result  of  explosive  rupture  of  railcar 
(source:  Florida  A&M  University) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Fluid  Dynamics  Simulations 

•  Typical  Applications: 

-  Air  flow  around  solid  shapes 

-  Hydrodynamic  analysis 


•  Examples: 

-  ANSYS  (www.ansys.com) 

-  HYB-3D  (University  of  Alabama^at  Birmingham) 


Flow  around  the  Space  Shuttle  (source:  NASA)  Chlorine  spill  dispersion  in  an  urban  area  (source:  UAB) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Materials  Engineering  Models 

•  Typical  Application: 

-  Predicting  fatigue  crack  growth  in  structures 

•  Example: 

-  AFGROW  (Air  Force  Growth) 


Example  of  crankshaft  fatigue  (source:  Wikipedia) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Printed  Circuit  Design  Simulations 

•  Typical  Applications: 

-  Simulation  of  electrical  circuit  board  behavior  during  design 

•  Examples: 

-  SPICE  (Simulation  Program  with  Integrated  Circuit  Emphasis) 


1973  Cal  Berkeley,  open  source,  spawned  commercial  variants 


Logisim  (digital  circuits  only,  open  source,  student  audience) 


Screen  shot  of  Logisim  2.3.4,  released  April  1 , 
2010  (source:  Hendrix  College  web  site) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Electrical  Power  System  Design  Simulations 

•  Typical  Applications: 

-  Simulation  of  power  systems  for  buildings  and  communities 

-  Simulation  of  a  regional  or  national  electric  power  grid 

•  Examples: 

-  eMEGAsim  -  OPAL-RT  Technologies 

-  RTDS®  (Real  Time  Digital  Simulator)  -  RTDS  Technologies 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Computer  Network  Engineering  Simulations 

•  Typical  Applications: 

-  Design  and  performance  evaluation  of  computer  networks 

-  Simulation  of  natural  and  man-made  network  disruptions 

•  Examples: 

-  OPNET  Modeler 

-  Joint  Communication  Simulation  System  (JCSS)  [formerly  NETWARS] 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Acoustic  Propagation  Models 

•  Typical  Applications: 

-  Determination  of  detection  ranges  for  underwater  sound  sources 

-  Determination  of  sound  speed  based  on  environmental  features 

•  Examples: 

-  Automated  Signal-Excess  Prediction  System  (ASEPS)  Transmission 
Loss  (ASTRAL) 

-  Modular  Ocean  Data  AssimilatiorJ^ System  (MODAS) 


ASTRAL  transmission  loss  curves  (source:  MODAS  surface  sound  speed  (source:  Biondo, 

Biondo  &  Mandelberg  -  MIV  project)  Mandelberg  et  al  -  JWARS-MIV  project) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Electromagnetic  Propagation  Models 

•  Typical  Application: 

-  Determination  of  atmospheric  detection  ranges  for  electromagnetic 
sources 

•  Example: 

-  Tropospheric  Electromagnetic  Parabolic  Equation  Routine  (TEMPER) 


TEMPER  3D  Result 


*  Propagation  factor  “F”  =  |  E  /  Ed  where  Eq  is  free-space  field 
Source:  Awadallah,  et  al,  Radar  Propagation  in  3D  Environments,  2004 
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Manufacturing  Process  Models 


•  Typical  Applications: 

-  Determining  and  optimizing  production  rates 

-  Determining  bottlenecks  in  planned  production  lines 


•  Examples: 

-  ExtendSim 

-  Arena 


Source:  Strickland,  Discrete  Event  Simulation  Using  Extend,  2009 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Integrating  Engineering-Level  Simulations 

•  The  process  of  integrating  engineering-level  simulations  is  similar  to  the 
process  of  integrating  a  system 

-  Each  simulation  acts  as  a  component  of  the  integrated  simulation  (often 
referred  to  as  a  “federation”  of  simulations) 

-  Data  interchange  agreements  for  simulations  are  like  interface  control 
documents  for  systems 

•  Engineering-level  simulations  can  liejlntegrated  with  one  another 

-  Through  sequential  passing  of  data  from  one  simulation  to  the  next 

■  Possible  if  there  are  no  significant  “feedback”  paths 

■  Can  be  done  through  automation,  or  manually  (a.k.a.  “sneaker  net”) 

-  Through  run-time  interoperability  (e.g.,  using  the  High  Level  Architecture 
for  simulation  interoperability,  IEEE  1516) 

■  Requires  pre-execution  agreements  as  to  which  simulations  “publish” 
and  “subscribe  to”  data  elements 

■  Requires  a  run-time  infrastructure  to  manage  execution 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Integrating  Engineering-Level  Simulations 
Through  Sequential  Data  Passing 


•  Need  to  establish  that  a  given  simulation  produces  outputs  that  are 
compatible  with  inputs  required  by  the  next  simulation  in  the  sequence,  either 
directly,  or  by  some  well-defined  transformation 

-  “Post-processing”  and/or  “pre-processing”  steps  may  be  required  to 
ensure  output-input  compatibility 

•  “Syntactic”  interoperability  -  refers  tqjnsuring  that  the  (post-processed)  data 
outputs  are  the  same  data  element  and  are  in  the  same  units  of  measure  as 
the  (pre-processed)  data  inputs 

•  “Semantic”  (or  “substantive”)  interoperability  -  refers  to  ensuring  that  the 
(post-processed)  data  outputs  and  (pre-processed)  data  inputs  have  the 
same  meaning  in  both  simulations 

-  For  example,  if  a  simulation  generating  “speed”  data  assumes  over-the- 
ground  speed  and  the  data-receiving  simulation  assumes  through-the-air 
speed,  there  is  no  semantic  interoperability 

•  Both  syntactic  and  semantic  interoperability  are  needed  for  two  simulations 
to  be  “composable” 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Integrating  Engineering-Level  Simulations 
Through  Run-Time  Interoperabiiity 


•  Most  engineering-level  simulations  require 

-  Causality:  The  effects  of  an  action  are  observed  after  the  action  occurs 

-  Repeatability:  The  simulation  gives  the  same  result  if  executed  twice 

•  Ensuring  causality  and  repeatability  requires  a  method  for  maintaining  “event 
ordering”  at  run-time 

-  This  is  a  non-trivial  problem  when  executing  several  simulations 
interactively  across  a  network,  in  which  packets  may  arrive  in  a  different 
order  from  the  order  in  which  they  were  generated 

•  Other  issues  for  simulations  interacting  at  run-time 

-  Maintaining  a  consistent  environment  (terrain,  weather)  over  time 

-  Deciding  which  simulation  should  have  control  of  an  entity  at  a  given  time 

-  Managing  the  number  of  interactions  required  (e.g.,  having  a  maximum 
range  for  a  sensor  so  not  every  sensor-target  pair  needs  to  be  evaluated) 

-  Detecting  that  another  simulation  has  stopped  executing 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Time  Management  in  Simulations 
Interoperating  at  Run-Time  -  Time  Definitions 


•  Wallclock  time  -  The  actual  time  of  day  during  a  simulation  execution  (e.g., 
today  from  4  pm  to  6  pm) 

•  Physical  time  -  The  time  in  the  physical  system  being  modeled  being 
modeled  by  the  simulation  (e.g.,  from  midnight  to  6  pm  on  December  7, 

1941) 

•  Simulation  time  (logical  time)  -  The  sinnulation’s  representation  of  physical 
time  (e.g.,  double-precision  floating  point  number  between  0  and  18,  where  a 
simulation  time  unit  represents  an  hour  of  physical  time) 

•  Federate  time  -  The  logical  (simulation)  time  within  a  particular  simulation 
federate  at  any  instant  during  a  distributed  simulation  execution 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

A  Logical  View  of  Time  Management 
(from  the  High  Level  Architecture) 


receive 

order 

messages 


FIFO 

queue 


time  stamp 
order 
messages 


TSO 

queue 


state  updates 
and  interactions 


Runtime 

Infrastructure 

(RTI) 


logicai  time  advance 
requests  and  grants 


federate 

•local  time  and  event  management 

•mechanism  to  pace  execution  with  wallclock  time  (if  needed) 
•federate-specific  techniques  (e.g.,  time  compensation) 


Wallclock  time 

(synchronized  with 
other  processors) 


from  Fujimoto  1998,  “Time  Management  in  the  High  Level  Architecture,”  Figure  2 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Module  Summary 


•  A  broad  range  of  engineering  disciplines  are  involved  in  design  and 
development,  thus  requiring  a  broad  range  of  models  and  simulations, 
primarily  at  the  engineering  level. 

•  A  systems  engineer  responsible  for  M&S  needs  to  have  a  broad  view  of  M&S 
tools  that  can  be  applied  in  a  range  of  disciplines  during  this  phase. 

•  The  process  of  integrating  engineering-level  simulations  is  similar  to  the 
process  of  integrating  a  system  1 

•  Engineering-level  simulations  can  be  integrated  with  one  another 

-  Through  sequential  passing  of  data  from  one  simulation  to  the  next 

-  Through  run-time  interoperability 

•  Both  syntactic  and  semantic  interoperability  are  needed  for  two  simulations 
to  be  composable 

•  Most  engineering-level  simulations  require  causality  and  repeatability 

•  Event  ordering  and  time  management  are  important  for  engineering-level 
simulations  with  run-time  interoperability  requirements 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Module  Objective  and  Outline 


Module  Objective:  To  describe  the  use  of  modeling  and  simulation  in  the 

integration  and  test  &  evaluation  phase  of  the  systems  engineering  process; 
and  to  describe  special  issues  particular  to  this  phase. 

Module  Outline 

•  Scope  of  Integration  and  Test  &  Evaluation  (T&E) 

•  A  Simplified  Process  Model  for  IntegratiorTand  T&E 

•  Simulation  Use  During  Integration  ■ 

•  Planning  for  Use  of  Models  and  Simulations  During  T&E 

•  Simulation  Use  During  Testing 

•  Post-Test  Evaluation  Using  Models  and  Simulations 

•  Summary 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Scope  of  Integration  and  Test  &  Evaluation  (T&E) 


•  The  integration  and  T&E  phase  of  systems  engineering,  as  discussed  in  this 
course,  corresponds  to  the  Integration  and  Evaluation  phase  in  the 
Kossiakoff  and  Sweet  textbook 

•  Integration  takes  unit-tested  components  and  subsystems  and  forms  them 
into  an  integrated  system 

•  Test  and  evaluation  (T&E)  of  military  systems  is  typically  divided  into 

-  Developmental  test  and  evaluation  (DT&E)  conducted  under  the 
auspices  of  the  system’s  program  manager 

-  Operational  test  and  evaluation  (OT&E)  conducted  by  an  independent 
operational  test  agency  (OTA) 

•  Integration  and  test  activities  are  typically  aided  by  live,  virtual,  and 
constructive  simulations  running  at  or  near  real  time 

•  Evaluation  activities  sometimes  involve  models  of  the  system  and  its 
components  to  aid  in  determining  the  source  of  unexpected  test  performance 


118 


IM 


Coolahan 
Associates,  LLC 


Modeling  and  Simulation  in  the  Systems  Engineering  Process 


A  Simplified  Process  Model  for 
Integration  and  T&E 


Deficiencies  Found  During  Test  Evaluation 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Simulation  in  Integration  -  Use  of  Stimulators 


•  As  one  proceeds  from  unit  testing  to  system  integration,  there  is  often 
a  need  for  “stimulators”  to  represent  a  part  of  the  system  (or  the 
external  environment)  that  is  not  currently  available  for  integration 

•  Examples  of  stimulators: 

-  Generation  of  an  infrared  (IR)_scene  to  be  sensed  by  an  IR 
seeker 

-  Representation  of  a  radar  (or  other  sensor)  output  as  it  would  be 
presented  to  its  processing  system 

-  Representation  of  a  potential  human  operator’s  input  to  a  vehicle 
control  system 

•  Gradually  substitute  real  system  components  for  simulated  system 
components  until  full  system  is  integrated 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Simulation  Issues  of  Particular  Interest 

During  Integration 


•  Representativeness  of  integration  environment  as  compared  to  the  intended 
operational  environment 

-  Are  characteristics  of  the  simulated  external  environment  sufficiently 
realistic,  in  terms  of  frequency,  intensity,  etc.? 

•  Real-time  operation  (often  “hard-real-time”) 

-  Can  the  software  simulation  of  ajiardware  component  operate  quickly 
enough? 

-  Can  simulation/stimulation  components  adequately  represent  the 
frequency  and  periodicity  of  the  real  system  components? 

•  Similarity  of  simulator/stimulator  interfaces  to  those  of  the  objective  system 
component 

-  Are  the  interfaces  of  the  simulator/stimulator  the  same  as  those  of  the 
system  component  being  represented?  Or  sufficiently  similar  so  that 
differences  can  be  accommodated  without  sacrificing  realism? 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Planning  for  Use  of  Models  and  Simulations 

During  T&E 


•  Need  to  determine  the  appropriate  integrated  combination  of  models, 
simulations,  and  test  events  to  obtain  the  most  credible  data  with  which  to 
conduct  a  comprehensive  evaluation  of  system  performance 

-  Are  there  situations  where  safety  precludes  testing? 

-  Are  there  physical  constraints  (e.g.,  size  of  test  range)? 

-  Are  there  fiscal  constraints  (e.g.,Jor  system  of  systems  testing)? 

•  Need  to  identify  areas  where  actual  luting  either  can  be  augmented  by  M&S 
or  used  to  validate  the  models  and  simulations 

•  Need  to  summarize  the  model,  simulation,  and  data  verification,  validation, 
and  accreditation  (VV&A)  to  be  conducted 

•  Need  to  document  how  the  integrated  use  of  accredited  models  and 
simulations  with  operational  testing  will  increase  the  knowledge  and 
understanding  of  the  capabilities  and  the  limitations  of  the  system  as  it  will  be 
employed 

•  Need  to  include  the  resources  required  to  perform  VV&A  of  the  models  and 
simulations;  to  obtain  and  maintain  the  models  and  simulations;  and  the 
resources  required  to  archive  data 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


The  Model-Test-Model  (MTM)  Paradigm 


•  The  Model-Test-Model  (MTM) 
paradigm  refers  to  the  iterative  use  of 
models  &  simulations  and  testing  to 
refine  the  modeled  representation  of  a 
system 

•  Start  with  a  best  estimate  of  the 

system’s  performance  as  represented^  ? 
in  a  model  or  simulation  ' 

•  Conduct  testing  on  the  (prototype) 
system  to  collect  data  on  how  the 
system  performs  in  reality 

•  Use  the  data  collected  to  refine  the 
modeled  representation  of  the 
system’s  performance 

•  Repeat  as  necessary  until  the  modeled 
representation  of  the  system  is 
deemed  adequate 


Start 


Finish 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


The  Simulation  Test  and  Evaluation  Process  (STEP)  -  a 
1990s  DoD  Attempt  at  Integrating  M&S  and  T&E 


Source:  Simulation,  Test,  and  Evaluation  Process  (STEP)  Guidelines,  Director,  Operational  Test  and  Evaluation, 
and  Director,  Test,  Systems  Engineering  and  Evaluation,  4  Dec  1997. 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Hardware-  and  Software-in-the-Loop 
Simulations  For  Testing 


•  Hardware-in-the-loop  (HWIL)  simulations  are  a  good  example  of  simulations 
that  are  generally  not  (or  need  not  be)  computer-based 

-  Examples: 

■  Wind  tunnels  for  missiles  and  aircraft 

■  Anechoic  chambers  for  radar  seekers 

■  Scene  generators  for  focal  plarie  arrays 

■  Tow  tanks  for  maritime  vehicles 

■  Pressure  chambers  for  submersible  vessels/housings 

■  Crash-test  facilities  for  automobiles 

■  Shake  tables  for  mechanical  structures 

■  Vacuum  chambers  for  spacecraft 

-  Require  calibrated  instrumentation  to  collect  data  on  the  system  under 
test 

•  Software-in-the-loop  (SWIL)  simulations  embed  actual  system  software  in  a 
synthetic  environment  representing  the  system’s  intended  use 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Examples  of  HWIL  Facilities 


NHTSA  crash  test  David  Taylor  Model  Basin,  Carderock 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Pre-Test  Predictions  Using  Models  and  Simulations 


•  In  modern-day  system  acquisition,  having  a  (perceived)  failure  during  a 
highly  visible  system  test  can  de-rail  the  system  development  process 

•  By  modeling  the  system’s  performance  in  the  test  environment  prior  to  the 
actual  test,  one  can 

-  Vary  environmental  parameters  to  determine  if  there  are  any  situations  in 
which  the  test  should  be  delayedj>ecause  of  excessive  risk  (e.g., 
extreme  wind  shear  conditions,  extreme  hot  or  cold  temperatures) 

-  Establish  an  “objective”  benchmark  with  which  to  compare  the  actual  test 
results 

■  Aids  in  determination  as  to  whether  the  test  was  “successful” 

-  Determine  boundaries  of  realistically  expected  performance,  for 
evaluating/ensuring  safety  during  the  test 

■  For  example,  “three-sigma”  ballistic  missile  trajectory  envelopes  for 
range-safety  decisions  on  whether  to  destroy  the  missile 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Simulation  Use  in  Test  Range  Activities 

•  Simulate  assets  (targets,  friendly  systems/platforms)  not  available  in 
the  range  environment  using  constructive  simulations 

-  For  large  scale  “system-of-systems”  tests  requiring  demonstration 
of  inter-system  interoperability 

•  Supplement  natural  environment_on  the  test  range  with  simulated 
natural  environment  features  not  present  on  the  test  range 

•  “Geo-relocate”  live  test  assets  from  other  test  ranges 
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TENA  Architecture  Overview 


TENA  Applications 


TENA  Tools 
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System  System 
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Source:  “Test  and  Training  Enabling  Architecture  (TENA)  Overview,”  Oct.  2015  (available  at  https://www.tena-sda.ora) 
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Joint  Mission  Environment  Test  Capability 
(JMETC)  Distributed  Test  Architecture 


Joint  Operational  Scenarios 
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Source:  “Test  and  Training  Enabling  Architecture  (TENA)  Overview,”  Oct.  2015  (available  at  https://www.tena-sda.ora) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Simulation  Issues  of  Particular  Interest 

During  Testing 


•  Latency  of  transmissions  across  the  network  of  constructive,  virtual,  and  live 
assets 

-  Need  to  maintain  representative  “real-time”  interactions 

•  Bandwidth  of  networks 


-  For  example,  environmental  data  often  needs  to  be  “pre-loaded”  because 
of  bandwidth  constraints 


|U 


.Jl 


•  Time  synchronization  among  geographically  distributed  systems 


-  GPS  time  source  often  used 


•  Consistency  of  environmental  representations  across  live,  virtual,  and 
constructive  simulation  assets 


•  Potential  safety  issues  introduced  by  adding  constructive  or  virtual  targets  to 
a  live  display 

-  For  example,  introducing  simulated  threat  aircraft  in  a  heads-up  cockpit 
display  could  result  in  evasive  maneuvers  into  a  real  mountain 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Post-Test  Evaluation  Using 
Models  and  Simulations  (1  of  2) 


•  Single-test  results 

-  Comparison  of  test  results  to  pre-test  model/simulation  predictions 

-  If  results  differ  from  predictions: 

■  Are  test  results  within  a  statistically-expected  range? 

■  Are  there  differences  in  the  day-of-test  environment  from  the 
predicted  environment?  -15;:, 

•  If  so,  can  do  a  “post-test  prediction”  based  on  the  day-of-test 
environment 

■  Does  test  data  indicate  an  obvious  anomaly  in  performance? 

-  If  differences  appear  to  be  “real”: 

■  Is  there  an  algorithmic  error  in  the  model/simulation? 

■  Is  there  an  un-modeled  effect  that  could  account  for  the  difference? 

■  Is  it  appropriate  to  “calibrate”  the  model/simulation  based  on  a  single 
test? 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Post-Test  Evaluation  Using 
Models  and  Simulations  (2  of  2) 


•  Multiple-test  results 

-  Comparison  of  multiple  test  results  to  pre-test  (or  post-test) 
model/simulation  predictions 

-  Unfortunately,  except  for  “high-value  systems”  (e.g.,  Navy  Trident  missile 
system),  it  is  seldom  possible  to  conduct  enough  full-system  tests  to  get 
statistically  significant  results  _ 

-  Is  there  a  pattern  (bias)  of  the  test  results  when  compared  to  the 
model/simulation  predictions?  If  so, 

■  Is  there  an  algorithmic  error  in  the  model/simulation? 

•  Can  use  of  statistical  modeling  techniques  (e.g.,  Kalman  filter) 
help  to  reveal  the  source  of  the  error? 

■  Is  there  an  unmodeled  effect  that  could  account  for  the  difference? 

■  Is  it  appropriate  to  “calibrate”  the  model/simulation  based  on  this 
number  of  tests? 
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Example  of  Multi-Test  Evaluation 

JHU/APL  Trident  II  Accuracy  Evaluation 

^ - -  - - 


Testing 
Test  1 


Per-test 

processing 


Cumulative 

model 

estimator 


Propagation 
over  scenarios 


Figure  5.  Model  estimation  for  Trident  II  resulting  in  the  credibie  performance  prediction 
of  a  criticai  system  to  the  government  and  miiitary  system.  =  true  model  parameter 
vector,  §  =  estimate  of  d,  =  covariance  of  estimation  error  in  0.) 

Source:  Levy,  L.J.,  “’The  Systems  Analysis,  Test,  and  Evaluation  of  Strategic  Systems,”  APL  Tech.  Digest,  Vol.  26, 
No.  4  (2005),  pp.  438-442. 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Module  Summary 


•  In  system  testing,  there  is  often  a  need  for  “stimulators”  to  represent  a  part  of 
the  system  (or  the  external  environment)  that  is  not  currently  available  for 
testing 

•  Use  of  models  and  simulations  during  T&E  must  be  planned  well  in  advance, 
in  conjunction  with  the  overall  T&E  plan 

•  Models  and  simulations  are  instrumental  in  pre-test  predictions  of  system 

performance  I' 

•  Simulations  are  essential  to  represent  assets  (threat  and  friendly)  not 
available  for  system  testing 

•  Models  of  the  system  under  test  and  its  components  are  useful  in 
determining  the  specific  source  of  differences  between  pre-test  predictions 
and  system  test  performance 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Module  Objective  and  Outline 

Module  Objective:  To  describe  the  use  of  modeling  and  simulation  in  the 

production  and  sustainment  phase  of  the  systems  engineering  process;  and 
to  describe  special  issues  particular  to  this  phase. 

Module  Outline 

•  Scope  of  Production  and  Sustainment 

•  A  Simplified  Process  Model  for  Production  and  Sustainment 

•  Planning  for  Use  of  Models  and  Simulations  During  Production 

•  Model  and  Simulation  Use  During  Production 

•  Model  and  Simulation  Use  During  Sustainment 

-  Systems  Operation  Simulations 

-  Reliability  Modeling 

-  Logistics  Simulations 

-  Ownership  Cost  Modeling 

•  Summary 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Scope  of  Production  and  Sustainment 


•  The  Production  and  Sustainment  phase  of  systems  engineering,  as 
discussed  in  this  course,  corresponds  to  the  Post-Development  Stage, 
consisting  of  the  Production  and  Operations  &  Support  phases,  in  Systems 
Engineering:  Principles  and  Practice  [1]. 

•  Production  takes  the  production  design  that  results  after  Test  &  Evaluation, 
and  “realizes”  one  or  more  instances  of  the  system 

-  Relatively  straightforward  for  software-only  systems 

-  Can  be  quite  complex  for  hardware  systems 

•  Sustainment,  which  includes  Operations  and  Support,  is  typically  the 
lengthiest  phase  for  a  system,  lasting  as  long  as  60  years  for  large-scale 
military  systems  (e.g.,  aircraft  carriers  and  the  B-52  bomber) 

-  Can  incur  up  to  50%  of  the  Total  Ownership  Cost  (TOC)  of  a  system 

-  Planning  for  Sustainment  (and  Disposal)  using  models  and  simulations 
needs  to  occur  early  in  the  systems  engineering  process 

Source:  (1)  Kossiakoff,  A.,  Sweet,  W.  N.,  Seymour,  S.  J.,  and  Biemer,  S.  N.,  Systems 

Engineering:  Principles  and  Practice,  Second  Edition,  John  Wiley  &  Sons,  Inc., 
Hoboken,  N.  J.  (2011). 
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Military  System  Total  Ownership  Cost 
by  Phase,  and  When  Determined 


0 

MS  0  MSI 


5 

MS  II 
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MS  III 
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Efficient  Exploration  of  the  Design  Space  Early  in 
the  Program  Is  Key  to  Reducing  Total  Ownership  Cost 


Source:  The  Simulation  Based  Acquisition  Vision:  A  Brief  Tutorial,  Nicholas  E.  Karangelen,  March  1998 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

A  Simplified  Process  Model  for 
Production  and  Sustainment 


Disposal 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Planning  for  Production 
Using  Models  and  Simulations 


•  Just  as  one  needs  to  plan  early  for  Test  and  Evaluation,  one  also 
needs  to  plan  early  for  Production,  particularly  for  hardware  systems 

-  What  rate  of  production  is  required? 

-  How  large  does  a  facility  (do  facilities)  need  to  be? 

-  What  is  a  good  production  process? 

•  Ensuring  that  computer-aided  design  (CAD)  models  of  the  system 
produced  during  Design  and  Development  can  flow  seamlessly  into 
computer-aided  manufacturing  (CAM)  equipment 

•  Modeling  the  design  of  production  facilities  (using  CAD) 

•  Simulating  the  flow  of  the  system  assembly  process  (using  process 
models,  such  as  Arena) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Model  and  Simulation  Use  During  Production 


•  CAD  models  of  the  system  produced  during  Design  and 
Deveiopment  are  ingested  by  CAM  equipment  to  automate 
component  manufacturing 

•  Modeis  of  production  manufacturing  faciiities  created  during  Design 
and  Deveiopment  are  refined,  based  on  the  production  design  of  the 
system 

•  Simuiations  of  the  fiow  of  the  manufacturing  process  are  executed, 
and  the  process  iterated 

-  To  optimize  the  assembiy  iine  itseif 

-  To  optimize  the  timing  of  the  fiow  of  component  parts  into  the 
system  assembly  facility 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Examples  of  Production  Facilities 


F-35  (Joint  Strike  Fighter)  production  facility 
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M&S  Standards  in  Production  -  Standard  for  the 


Exchange  of  Product  Model  Data  (STEP) 


AP  203:  Configuration  Controlled  3D 
Designs  of  Mechanical  Parts  and 

Assemblies  poEsino. 


Configuration  Management 

*  Authorisation 

*  Control  (Version/Revision) 

^  Eff activity 

L  hJ*  \ 

Geometric  Shapes 
^  Advanced  BREP  Solids 

*  Faceted  BREP  Solids 

*  Manifoid  Surfaces  with  Topology 

^  Release  Status 

*  Wireframe  with  Topology 

^  Security  Classification 

*  Surfaces  and  Wireframe  without 

^  Supplier 

Topology 

Product  Structure 

^  Assemblies 
^  Bill  of  Materials 
^  Part 

^  Substitute  Part 
*  Alternate  Part 


Specifications 

^Surface  Finish 
^  Matenal 

*  Design 

*  Process 

*  CAD  Filename 


•  Boeing  Commercial  Aircraft 

•  Boeing  CSTAR 

•  Delphi  Automotive  Systems 

•  Lockheed  Martin 

•  NASA 


Source:  Manufacturing  Interoperability  &  the  Manufacturing  Systems 
Integration  Division,  Steven  Ray,  Ph.D.,  National  Institute  of  Standards 
and  Technology,  May  1 1 , 2001 
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M&S  Standards  in  Production  -  Core 
Manufacturing  Simulation  Data  (CMSD)  Standard 


f]  "ii 


r 


Approved  as  a  Simulation  Interoperability  Standards  Organization  (SISO) 
standard,  spring  2010 

Utilizes  Unified  Modeling  Language  (UML)  class  and  package  diagrams 

CMSD  information  categories: 

-  Calendar  information 

-  Resource  information 

-  Skill  information 

-  Setup  information 

-  Part  information 

-  Bill-of-materials  information 

-  Inventory  information 

-  Process  plan  information 

-  Maintenance  plan  information 

-  Order  and  Job  information 

-  Schedule  information 

-  Reference  information 

-  Probability  distribution  information 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Model  and  Simulation  Use  During  Sustainment 


•  Operations  of  the  system  are  simulated  under  controlied  conditions  to 
reproduce  system  failures  experienced  in  the  operationai 
environment,  and  to  investigate  potential  solutions 

•  Reiiabiiity,  Avaiiabiiity,  and  Maintainabiiity  of  the  system  are  modeled 
and  re-modeled  periodically,  using  data  from  systems  in  the 
operationai  environment  -S 

•  Logistics  for  the  repair  and  supply/re-suppiy  of  spare  parts  for  the 
system  are  simuiated 

•  Ownership  costs  are  modeied  on  a  continuing  basis 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Systems  Operation  Simulations 


•  Simulators  replicating,  as  closely  as  possible,  the  system  or  major 
subsystems  thereof,  are  often  operated  and  maintained  for  high- 
value  and  high-volume  systems 

•  Examples 

-  Simulators  for  systems  opera^ng  in  a  remote  environment  (e.g., 
system  work-arounds  for  Apollo  13,  unmanned  interplanetary 
spacecraft) 

-  Subsystem  simulators  to  investigate  infrequent  operational 
problems  (e.g.,  reported  anomalous  auto  acceleration  events) 

-  Simulations  of  system  component  failures  for  accident  forensics 
(e.g.,  space  shuttle  wing  penetration  by  foam  during  launch) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Reliability,  Availability,  and  Maintainability  (RAM) 


•  Reliability  -  the  probability  that  a  system  will  perform  its  function  correctly  for 
a  specified  period  of  time  under  specified  conditions 

-  Typical  metric:  Mean  Time  Between  Failure  (MTBF) 

•  Maintainability  -  a  measure  of  the  ease  of  accomplishing  the  functions 
required  to  maintain  a  system  in  a  fully  operable  condition 

-  Typical  metric:  Mean  Time  To  Repair  (MTTR) 

•  Availability  -  the  probability  that  a  system  will  perform  its  function  correctly 
when  called  upon 

-  Typical  metric:  Probability  of  availability  (Pa) 

-  Pa  ®  1  -  MTTR  /  MTBF  (for  short  repair  times  and  low  failure  rates) 

-  Note:  Operational  availability  (AJ  is  often  used  as  a  data  element  in 
military  campaign  simulations 


Source  of  definitions:  Systems  Engineering:  Principies  and  Practice,  Second  Edition,  Chapter  12 
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Reliability  Modeling 


•  The  reliability  of  a  system  can  be  modeled  as  a  mathematical  function  of  the 
reliability  of  its  components 

-  For  a  system  of  10  critical  independent  non-redundant  components, 

PR  =  P,lXP,2X---XPr10 

-  For  a  system  with  two  independent  redundant  components  with  failure 
probabilities  Pf^  and  Pf2, 

Pr  =  1  -  Pf1  X  P,2 

•  For  a  major  system  with  many  subsystems  and  components,  the  reliability 
model  can  become  quite  complicated,  and  is  very  dependent  on  accurate 
estimates  of  component  reliabilities 

•  Example:  Idaho  National  Laboratory  (INL)  SAPHIRE 
(Systems  Analysis  Programs  for  Hands-on  Integrated 
Reliability  Evaluations) 

-  Implements  Probabilistic  Risk  Assessment  (PRA) 

-  Used  by  NRC  and  NASA 
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Repair  and  Spare  Parts  Logistics  Simulations 


Similar  to  supply  chain  simulation  during  production  of  a  system 

Essentially  a  process  simulation  tailored  to  the  repair  and  supply/re-supply  of 
spare  parts  for  system  support 

Various  process  modeling  tools  can  be  used 

-  Arena 

-  ExtendSim 

-  AnyLogic 


Example  logistics-specific  models  and  simulations 

-  Supply-Chain  Operations  Reference  (SCOR)  model 

-  U.S.  Air  Force  Logistics  Simulation  (LOGSIM) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Ownership  Cost  Modeling 

•  Need  to  include  all  costs  associated  with  continued  ownership  of  a  system 

-  Personnel  (operations  and  maintenance) 

-  Fuel  /  power 

-  Repairs  and  spare  parts 

•  A  variety  of  ownership  cost  models  exist 

-  ACEIT  (Automated  Cost  Estimating  Integrated  Tools) 

-  SEER-H  (hardware),  SEER-SEM  (software)  [Galorath] 

-  Automotive  System  Cost  Modeling  (ASCM)  Tool  [Oak  Ridge] 

-  Cost  Analysis  Strategy  Assessment  (CASA)  [US  Army  LEC] 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Module  Summary 


•  Production  of  a  hardware  system  must  be  planned  well  in  advance, 
using  models  and  simulations  of  facilities  and  processes 

•  Sustainment  (operations  and  support)  costs  are  usually  the  largest 
element  of  the  ownership  cost  for  major  military  systems 

•  Progress  is  being  made  in  the  de^yelopment  of  standards  for  models 
and  simulations  used  for  produciioh 

•  System  operation  simulations  are  useful  for  troubleshooting  problems 
with  systems  operating  in  a  remote  environment 

•  Process  modeling  tools  are  important  for  both  production  and 
sustainment 

•  Reliability  models  can  be  quite  complex  for  major  systems 

•  System  cost  models  need  to  consider  the  cost  of  all  elements 
associated  with  the  ownership  of  a  system 


168 


Coolahan  Modeling  and  Simulation  in  the  Systems  Engineering  Process 

AssociAtts^  LLC 

Typical  Simulation  Resolution  Levels  During 
Phases  of  the  Systems  Engineering  Process 


Campaign 

Mission 

Engagement 

Engineering 

Needs  /  Concept  Exploration  Design  /  Integration  /  Production  / 

Opportunities  /Evaluation  Development  T&E  Sustainment 

IM 


Coolahan 
Associates,  LLC 


Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Typical  Simulation  Resolution  Levels  During 
Phases  of  the  Systems  Engineering  Process 


Campaign 


Mission 


Engagement 


Engineering 


Needs  /  Concept  Exploration  Design  /  Integration  /  Production  / 
Opportunities  /Evaluation  Development  T&E  Sustainment 
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Selected  Detailed  Examples  (as  time  permits) 


System  Effectiveness  Simulation  Examples 

-  Conceptual  model  for  a  communications  system 

-  Logical  data  model  for  a  scenario 
Interacting  Simulation  Examples 

-  A  Crisis  Management  and  Evacuation  System 

-  A  Mobile  Missile  System 
Integration  and  T&E  Examples 

-  Construction  of  a  Simulation  Environment  for  an  Underwater 
Vehicle’s  Navigation  and  Sensor  Data  Systems 

-  Construction  of  the  M&S  Portions  of  a  Test  and  Evaluation  Master 
Plan  (TEMP) 

Repair  Process  for  a  Deployed  Military  System  Component 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


System  Effectiveness  Simulation  Example  - 
Conceptual  model  for  a  communications  system  (1  of  4) 


•  Question  to  be  answered  -  how  effective  would  a  new  radio  frequency 
communications  system  be  in  a  varied-terrain  environment,  in  the  possible 
presence  of  rain,  with  the  possibility  of  jamming  by  an  adversary? 

•  Develop  a  simulation  conceptual  model  in  graphical  form 

•  What  modeling  and  simulation  components/elements  are  required? 

-  Digital  Terrain  Elevation  Data  (DT^E^)  ^^r  area  of  interest 

-  Initial  location  and  movement  scripts  for  source,  receiver,  and  jammer 

-  Rain  movement  as  a  function  of  time 

-  Probability  of  successful  communication  vs.  distance  in  a  benign  line-of- 
sight  environment 

-  Degradation  of  probability  of  successful  communication  as  a  function  of: 

■  Distance  of  propagation  through  rain 

■  Distance  and  azimuth  of  jammer  relative  to  source  and  receiver 

-  Other? 
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System  Effectiveness  Simulation  Example  - 
Conceptual  model  for  a  communications  system  (2  of  4) 


Measure  of  effectiveness 

-  Probability  of  successful  receipt  of  a  message  in  a  (set  of)  representative 
operational  environment(s) 

The  system  representation  (in  performance  terms) 

-  Source  characteristics  (frequency  range,  power  levels,  directionality) 

-  Receiver  characteristics  (frequency  range,  sensitivity,  directionality) 

The  system’s  concept  of  operations  I 

-  Rules  on  variations  in  power  level  selections  and  antenna  pointing  angle 
by  operator 

The  representation  of  threats  and  friendly  systems 

-  Jammer  source  characteristics  (frequency  range,  power  levels, 
directionality) 

The  representation  of  the  natural  and  man-made  environment 

-  DTED  data  (level  2) 

-  Rain  effects  (attenuation  by  frequency  range  and  rain  density) 
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System  Effectiveness  Simulation  Example  - 
Conceptual  model  for  a  communications  system  (3  of  4) 

•  The  scenario 

-  Movement  scripts  for  source,  receiver,  and  jammer 

-  Rain  density,  expanse,  and  movement  vs.  time 


Transmitter 


Receiver 


Terrain  profile 
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System  Effectiveness  Simulation  Example  - 
Conceptual  model  for  a  communications  system  (4  of  4) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

System  Effectiveness  Simulation  Example  - 
Logical  data  model  for  a  scenario 


•  A  time  of  year  and  duration 

•  Location  and  extent  of  the  play  box(es) 

-  Example:  coordinate  sets 

•  Instantiations  of  the  natural  and/or  man-made  environment 

-  Example:  environment  sets 

•  The  numbers  and  types  of  assets  (system-of-interest,  friendly,  threat, 

neutral)  t 

•  System  concepts  of  operation,  and  the  way  in  which  assets  move 

-  Example:  scripted  way  points 
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System  Effectiveness  Simulation  Example  - 
Logical  data  model  for  a  scenario  -  Scenario  identification 


•  Scenario  ID 

•  Title 

•  Objective 

•  Author 

•  Date 

•  Start  time  (GMT) 

•  End  time  (GMT) 

•  Time  step 
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System  Effectiveness  Simulation  Example  - 
Logical  data  model  for  a  scenario  -  Coordinate  sets 


Coordinate  sets  may  be  expressed  as  multiple  X-Y-Z  or  Lat-Lon-Alt  points, 
reference  frame,  to  define  an  area  of  interest  (e.g.,  DIED  region,  play  box. 

Coordinate  set  ID 

Coordinate  set  type  (X-Y-Z  or  Lat-Lon-Alt) 

Reference  frame  (e.g.,  WGS  1984,  UTM) 

Number  of  coordinate  points 
For  coordinate  sets  of  type  X-Y-Z: 

-  Units 

-  For  each  coordinate  point: 


f]  -ii 


r 


in  some 
etc.) 


■  X 

■  Y 

■  Z 


For  coordinate  sets  of  type  Lat-Lon-Alt: 

-  Lat-Lon  units 

-  Alt  units 

-  For  each  coordinate  point: 

■  Lat 

■  Lon 


■  Alt 
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System  Effectiveness  Simulation  Example  - 
Logical  data  model  for  a  scenario  -  Environment  sets 


•  Environment  sets  can  used  to  describe  the  environment  (land,  air,  sea)  in  an 
area  of  interest 

•  Environment  ID 

•  Coordinate  set  ID  reference 

•  For  air  environments: 

-  Air  parameters  (e.g.,  cloud  cover^density) 

•  For  sea  environments:  '9 

-  Sea  parameters  (e.g.,  sea  state) 

•  For  land  environments: 

-  Land  parameters  (e.g.,  terrain  height) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

System  Effectiveness  Simulation  Example  - 
Logical  data  model  for  a  scenario  -  Assets 


Assets  may  be  of  a  number  of  different  types,  and  may  be  in  alliances  with 
other  assets,  with  the  alliances  related  as  friendly,  hostile,  or  neutral 

For  each  asset: 

-  Asset  ID 

-  Asset  classification  (e.g.,  vehicle,  command  post,  sensor) 

-  Asset  category,  within  classification  (e.g.,  ship,  radar) 

-  Alliance  ID  reference  Jr 

For  each  alliance: 

-  Alliance  ID 

-  Alliance  name 

-  Alliance  asset  IDs 

Alliance  relationships  -  for  each  relationship: 

-  Alliance  type  (friendly,  hostile,  or  neutral) 

-  “Subject”  alliance  ID 

-  “Predicate”  alliance  ID 
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System  Effectiveness  Simulation  Example  - 
Logical  data  model  for  a  scenario  -  Asset  movement 


•  Asset  movement  in  a  scenario  may  be  scripted,  by  specifying  a  series  of 
way-points  and  times,  or  by  specifying  a  series  of  courses,  speeds,  and 
durations 

•  Way-point  movement  plan  -  for  each  movement: 

-  Current  coordinate  set  ID  reference 

-  Next  coordinate  set  ID  reference^. 

-  Arrival  time  at  next  coordinate  set  (assume  constant  course  and  speed) 

•  Course-speed-duration  movement  plan  -  for  each  movement: 

-  Course  for  movement  (assume  constant) 

-  Speed  for  movement  (assume  constant) 

-  Duration  of  movement 
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Example  of  Scenario,  Play  Boxes,  Environment  Sets, 
and  Coordinate  Sets  Relationships 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Example  of  Asset  Relationships 


Allied  With 


Note:  Diagram  notation  is  IDEF1X, 
IEEE  Std  1320.2-1998. 


Source:  “A  Logical  Data  Model  and 
Translation  Software  for 
Scenario  Representations  in  Mission-Levei 
Simulations,”  J.  F.  Schloman,  2005  Spring 
Simulation  Interoperabiiity  Workshop,  San 
Diego,  CA  (Mar  2005) 
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Example:  Interacting  Simulations  for  a  Crisis 
Management  and  Evacuation  System 


Design  layout  of  a  chemical  sensor  system  for  a  downtown  urban  area,  and 
a  traffic  management  system  for  evacuation  during  a  crisis 

Component  Simulations 

-  Explosive  detonation  causing  railcar  rupture 

-  Chemical  source  strength  simulation 

-  Chemical  plume  dispersion  simulation 


-  Chemical  sensor  simulation  w 

-  Emergency  management  command  and  control  simulation 

-  Traffic  flow  simulation 
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Interacting  Simulations  for  a  Crisis  Management 
and  Evacuation  System  -  Scenario  Use  Case 


5.  Emergency 
responders  react 


4.  Chlorine  cloud 
moves  toward 
downtown 


3.  Second 
explosion,  15 
minutes  later 


2.  First  explosion 

1 .  Train  with 
railcars  containing 
chlorine 
approaches 
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6.  News  reports 
issued 


7.  Local 

commanders  order 
evacuation 


8.  Police  in  protective 
gear  dispatched  to 
intersections 


9.  Chemical  sensors 
deployed 


10.  Local  populace 
reacts,  traffic  builds 
on  roads 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Interacting  Simulations  for  a  Crisis  Management 
and  Evacuation  System  -  Design  Considerations 


•  Railcar  rupture  simulation  component 

-  Needs  no  feedback  from  other  simulation  components 

-  Can  be  executed  in  advance 

•  Simulation  of  airborne  transport  through  3D  cityscape 

-  Requires  many  processors,  cannot  run  in  real  time  -  3  steps: 

■  Generation  of  wind  field  (slower  than  real  time) 

■  Insertion  of  pollutant  into  wind  field  (slower  than  real  time),  forming  data 
file  of  chlorine  concentrations 

■  Extraction  of  chlorine  concentrations  in  real  time  from  data  file 

•  Airborne  transport  depends  on  release  rate  of  chlorine 

-  So  chlorine  release  simulation,  although  not  computationally  intensive,  needs 
to  be  executed  in  advance 

•  Remaining  three  functions  (sensing,  command  and  control,  and  traffic 
flow)  can  be  performed  in  real  time  (or  faster)  as  part  of  simulation 
federation 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Interacting  Simulations  for  a  Crisis  Management 
and  Evacuation  System  -  Block  Diagram 


Non-Real-Time  Simulation  Components  Real-Time  Simulation  Federation 

Components  -  Federated  Using 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Example:  Interacting  Simulations  for 
a  (Mobiie)  Missile  System 


•  Simulations  of  Interest 

-  Transporter-Erector-Launcher - 
Structural  Mechanics 

-  Missile  structure  -  Structural  Mechanics 
(During  Transport  and  Flight) 

-  Propulsion  -  Thrust,  Heat  Generation 

-  Thermal  -  Heat  Transfer  to  Nozzle  and 
Missile  Structure 

-  Guidance  and  control  -  6-dof  Flight 
Simulation 

-  Fluid  dynamics  -  Vane  Control 
Effectiveness 


Pershing  1 A  missile 
(Source:  U.S.  Army) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Interacting  Simulations  for  a  (Mobile)  Missile  System: 
Step  1 :  Where  might  there  be  interactions? 


Potential 

structural 

burn- 

through 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Interacting  Simulations  for  a  (Mobile)  Missile  System: 
Step  2:  Are  the  interactions  one-way  or  two-way?  (1  of  4) 


Interactions  between 
Propulsion  and  Guidance 
and  control  are  one-way, 
during  each  missile  stage’s 
burn  time. 

Thermal  heat  transfer  to 
Missile  structure  and  Vane 
control  to  Missile  structure 
are  one-way. 

For  these,  simulations  of 
the  first  can  be  run  to 
completion,  and  their 
outputs  input  to 
simulations  of  the  second. 
(“Batch  runs”  can  be 
used.) 


Potential 

structural 

burn- 

through 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Interacting  Simulations  for  a  (Mobile)  Missile  System: 
Step  2:  Are  the  interactions  one-way  or  two-way?  (2  of  4) 


TEL  Structural 

Transport  dynamics 

V. 

Missile  structural 

mechanics 

Erector  dynamics 

mechanics 

•  Pre-launch  dynamics  between  the  TEL  and  the  missile  are  two-way: 


-  During  transport,  the  missile  and  TEL  cradle  interact  in  a  relatively 
static  configuration 


-  When  the  erector  is  activated,  the  missile  and  TEL  erector  cradle 
interact  dynamically 


•  As  the  concern  is  structural  mechanics  for  both  the  missile  structure  and 
the  TEL,  a  unified  (tightly  coupled)  structural  mechanics  simulation  of  both 
can  be  constructed 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Interacting  Simulations  for  a  (Mobile)  Missile  System: 
Step  2:  Are  the  interactions  one-way  or  two-way?  (3  of  4) 


•  The  interactions  between  Guidance  and  control  and  Fluid  dynamics  of  vane 
movements  are  two-way 

-  Vane  position  commands  cause  vane  movement 

-  Vane  movement  produces  rotational  forces  on  the  missile 

•  Usually,  simulations  (computational  fluid  dynamics  codes  or  wind  tunnel 
tests)  are  run  in  advance  to  calculatejotational  forces  as  a  function  of  vane 
position,  missile  angle  of  attack,  an^  relative  velocity 

-  This  permits  the  calculation  of  rotational  forces  to  be  embedded  in  the 
Guidance  and  control  simulation 

•  For  complex  interactions,  the  Guidance  and  control  and  Fluid  dynamics  of 
vane  movement  could  be  in  separate  simulations  that  interchange  data 
during  run-time 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Interacting  Simulations  for  a  (Mobile)  Missile  System: 
A  few  basics  on  solid  rocket  propulsion  and  nozzles 


•  After  ignition,  solid  fuel  burns  radially  out  from  center  toward  motor  casing 

•  Fuel  burn  creates  hot  gases  that  exit  through  nozzle,  creating  thrust 

•  Thrust  depends  on  many  factors,  including  nozzle  throat  area 

•  Nozzle  lining  ablates  over  time,  slightly  increasing  nozzle  throat  area 
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Solid  rocket  motor  thrust  equations  (source:  NASA) 


Solid  rocket  motor  (source:  Wikipedia  Commons) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Interacting  Simulations  for  a  (Mobile)  Missile  System: 
Step  2:  Are  the  interactions  one-way  or  two-way?  (4  of  4) 


•  The  interactions  between  Propulsion  and  Thermal  heat  transfer  are  two-way, 
because  exit  gas  temperature  causes  ablation  at  nozzle  throat 

•  Because  of  complexity  of  interactions,  for  detailed  calculations  of  thrust  vs. 
time,  would  want  to  have  Propulsion  and  Thermal  heat  transfer  simulations 
interact  at  run-time 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Example:  Construction  of  a  Simulation  Environment  for  an 
Underwater  Vehicle’s  Navigation  and  Sensor  Data  Systems 


•  Consider  the  integration  of  a  tethered  underwater  vehicle’s  navigation  and 
sensor  systems 

•  The  vehicle  will  include: 

-  A  forward-looking  obstacle-avoidance  sonar 

-  Two  side-scan  sonars  (one  looking  left,  one  looking  right) 

-  Two  downward-looking  full-motion  video  cameras 

-  One  downward-looking  high-resolution  electronic  still  camera 

-  A  four-head  downward-looking  Doppler  sonar  for  navigation 

•  Prior  to  receiving  the  above  imaging  and  navigation  sensors,  how  could 
simulations  be  used  (as  stimulators)  to  prepare  for  the  sensors’  integration 
with  the  vehicle’s  navigation  and  sensor  data  acquisition  systems? 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Simulation  Environment  for  an  Underwater  Vehicle’s  Navigation 
and  Sensor  Data  Systems  -  Potential  System  Design 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Simulation  Environment  for  an  Underwater  Vehicle’s 
Navigation  and  Sensor  Data  Systems  -  Considerations 


•  In  the  system  being  built,  what  are  the  interfaces  between  the  imaging  /  navigation 
sensors  and  the  vehicle’s  navigation  and  sensor  data  acquisition  systems? 

-  Are  the  interfaces  analog  or  digital? 

-  For  analog  interfaces,  what  analog  data  communication  standards  are  being  used 
(video,  acoustic,  other)? 

-  For  digital  interfaces: 

■  What  digital  data  communication  hardware  standards  are  being  used  (e.g., 
RS-232,  Ethernet,  USB)?  C 

■  What  data  formatting  techniques  are  being  used  (e.g.,  XML,  byte-ordering 
scheme,  proprietary)? 

■  What  syntax  is  being  used  for  the  data  in  each  data  transmission  frame? 

■  What  is  the  frame  transmission  rate? 

•  To  what  degree  does  testing  require  that  simulated  data  be  representative  of  expected 
real  data? 

-  Are  only  the  data  rate  and  data  format/syntax  important? 

-  Do  images  need  to  be  realistic  (e.g.,  if  the  data  acquisition  system  employs 
feature  recognition  to  make  a  decision)? 

-  Does  navigation  sensor  data  need  to  be  used  to  develop  a  simulated  track? 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Simulation  Environment  for  an  Underwater  Vehicle’s 
Navigation  and  Sensor  Data  Systems  -  Video  Cameras 


•  Design  note:  Analog  video  camera  signals  are  merely  re-transmitted  in 
analog  form  to  the  surface  vehicle  for  viewing  by  operators  and  possible 
recording. 

•  Therefore  the  video  camera  simulations  (stimulators)  can  be  simple 
hardware  video  sources,  even  VCRs  with  arbitrary  interfaces 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Simulation  Environment  for  an  Underwater  Vehicle’s 
Navigation  and  Sensor  Data  Systems  -  Side-scan  Sonars 


•  Design  note:  Each  side-scan  sonar  produces  a  “line”  of  1024  pixels  with 
black-and-white  intensity  from  0  to  255,  once  per  second;  lines  are  merely 
re-transmitted  to  the  surface  vehicle. 

•  Therefore  the  side-scan  sonar  simulations  (stimulators)  need  only  replicate 
the  data  rates  of  the  sensors. 
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Simulation  Environment  for  an  Underwater  Vehicle’s  Navigation 
and  Sensor  Data  Systems  -  Obstacle  Avoidance  Sonar 


•  Design  notes: 

-  The  obstacle  avoidance  sonar  produces  a  “vertical  line”  of  256  pixels 
(covering  a  30-degree  vertical  field  of  view)  with  black-and-white  intensity 
from  0  to  255,  5  times  per  second,  sweeping  a  30-degree  horizontal  field 
of  view  in  30  seconds  to  form  a  256x150  continually-updated  image. 

-  The  data  acquisition  system  generates  an  alarm  when  a  “dark  object”  of 
a  certain  size  is  in  the  center  of  the  field  of  view. 


•  Therefore  the  obstacle-avoidance 
sonar  simulation  (stimulator)  must 
provide,  at  the  required  rate, 
representative  data  that  will  show 
both  no  dark  objects  and  an 
occasional  realistic  dark  object  over 
time. 


Obstacle- 

Electronic 

avoidance  sonar 

still  camera 

Digital  (vertical  line  scan) 


Bi-directional  digital  (command /image) 


1 

Port  sic 

so 

1 

.lt7-OOCll  1  1 

nar 

Video 
camera  1 

Analog 


Data  acquisition 
system 


Vehicle  navigation 
system 


Digital 


Dicital 


Analog 


Starboard  side- 
scan  sonar 


Video 
camera  2 


Four-head  Doppler 

Surface 

navigation  sonar 

7^ 

vehicle 

158 


Coolahan  Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Associates,  LLC 


Simulation  Environment  for  an  Underwater  Vehicle’s 
Navigation  and  Sensor  Data  Systems  -  Electronic  Still  Camera 


•  Design  note:  The  electronic  still  camera,  upon  a  command  from  the  data 
acquisition  system,  takes  a  single  1024  x  1024  pixel  (with  black-and-white 
intensity  from  0  to  255),  at  a  maximum  rate  of  once  per  second.  Images  are 
merely  re-transmitted  to  the  surface  vehicle. 


Therefore  the  electronic  still  camera  simulation  (stimulator)  needs  to  replicate 
the  data  rate  (up  to  8  megabits  per  second)  and  pixel  transmission  order  of 
the  camera,  upon  receipt  of  a  comi 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Simulation  Environment  for  an  Underwater  Vehicle’s 
Navigation  and  Sensor  Data  Systems  -  Doppler  Nav  Sonar 


•  Design  note:  The  Doppler  navigation  sonar  transmits  four  digital  values 
(from  0  to  4095),  once  per  second,  representing  fore,  port,  aft,  and  starboard 
speeds  relative  to  the  bottom  of  the  body  of  water.  The  vehicle  navigation 
system  uses  these  values  to  compute  an  instantaneous  vehicle  velocity  and 
to  produce  a  continuous  x-y  track  relative  to  the  bottom. 


Therefore  the  Doppler  navigation  ,, 
sonar  simulation  (stimulator)  needs 
to  provide  an  operationally  realistic 
(within  vehicle  propulsion 
capabilities),  time-consistent 
(second-to-second)  set  of  four 
speed  values  to  the  vehicle 
navigation  system). 
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Example:  Construction  of  the  M&S  Portions  of  a  Test  and 

Evaluation  Master  Plan  (TEMP) 


•  Consider  a  Test  and  Evaluation  Master  Plan  for  a  ballistic  missile  interceptor 
missile,  which  could  include  descriptions  of  such  T&E  activities  as 

-  Subsystem  tests  of  radar  seeker 

-  Subsystem  tests  of  focal  plane  array 

-  Wind  tunnel  tests  using  scaled  missile  model 


-  Static  tests  (on  test  stand)  of  propulsion  subsystem 

-  Flight  tests  on  a  test  range 

-  Post-flight  evaluation 

What  types  of  models  and  simulations  are  needed  for  each  T&E  activity? 

-  Where  might  simulations  be  used?  Where  might  models  be  used? 

-  For  the  simulations,  which  are  live?  Which  are  virtual?  Which  are 
constructive? 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

Extracts  from  DoD  Instruction  5000.02  Regarding 
Test  and  Evaluation  Master  Plan  (TEMP) 


•  Test  and  Evaluation  Master  Plan  (TEMP).  ...  The  TEMP  shall 
describe  planned  developmental,  operational,  and  live-fire  testing, 
including  measures  to  evaluate  the  performance  of  the  system  during 
these  test  periods;  an  integrated  test  schedule;  and  the  resource 
requirements  to  accomplish  the  planned  testing. ... 

-  (6)  Appropriate  use  of  accredited  models  and  simulation  shall 
support  DT&E,  lOT&E,  and  ifFT&E. 


DT&E:  Developmental  Test  &  Evaluation 
OT&E:  Operational  Test  &  Evaluation 
LFT&E:  Live  Fire  Test  and  Evaluation 


Source:  DoD  Instruction  5000.02,  Operation  of  the  Defense  Acquisition 
System,  December  8,  2008 
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References  to  Modeling  and  Simulation  in 
Recommended  TEMP  Format 

•  PART  Ml  -  TEST  AND  EVALUATION  STRATEGY 


-  3.3 

DEVELOPMENTAL  EVALUATION  APPROACH 

■  3.3.3 

Modeling  and  Simulation 

-  3.4 

LIVE  FIRE  EVALUATION  APPROACH 

■  3.4.2 

Modeling  and  Simulation 

-  3.6 

OPERATIONAL  EVALUATION  APPROACH 

■  3.6.2 

Modeling  and  Simulation 

•  PART  IV  -  RESOURCE  SUMMARY 

-  4.1  INTRODUCTION 

■  4.1.7  Models,  Simulations,  and  Test-beds 


Source:  Annex  to  Defense  Acquisition  Guidebook,  Section  9.10,  “Test 
and  Evaluation  Master  Plan  (TEMP)  Recommended  Format” 
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Developmental  Test  &  Evaluation: 
Tests  of  Interceptor  Sensor  Subsystems 


•  Radar  seeker  subsystem  testing 

-  Intended  to  estimate  performance  of  the  in-development  seeker 

-  Employs  hardware-in-the-loop  (HWIL)  simulation 

■  Radar  seeker  is  a  live  simulation  component  (the  real  seeker) 

■  Target  object  in  an  anechoic  chamber  is  a  constructive  simulation 
component  (a  simulation  of  a  potential  target) 

•  Focal  plane  array  subsystem  testing^ 

-  Intended  to  estimate  performance  of  the  in-development  array 

-  Employs  HWIL  simulation 

■  Focal  plane  array  is  a  live  simulation  component  (the  real  array) 

■  Target  representation  is  a  constructive  simulation  component  (e.g., 
an  array  of  light-emitting  devices  representing  various  target  and 
background  signatures) 

-  May  also  have  a  software-in-the-loop  (SWIL)  component 

■  Image  processing  software  embedded  in  seeker  system  for  target 
recognition  and  discrimination 
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Developmental  Test  &  Evaluation: 
Aerodynamic  and  Propulsion  Testing 


•  Wind  tunnel  testing  using  scaled  missile  model 

-  Intended  to  estimate  aerodynamic  performance  of  missile  at  various 
speeds  and  angles  of  attack 

-  Employs  a  physical  model  of  the  missile  body 

-  Wind  tunnel  test  itself  is  a  simulation 

■  Wind  field  is  a  constructive  environmental  simulation  component  (of 
the  real  relative  wind  the  missile  would  see  during  actual  flight) 

•  Static  testing  (on  test  stand)  of  propulsion  subsystem 

-  Intended  to  estimate  thrust  vs.  time  of  the  missile  interceptor 

-  Employs  HWIL  simulation 

■  Missile  stage  containing  propellant  and  ignition  system  is  a  live 
simulation  component  (the  real  missile  stage) 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 

DT&E  and  OT&E  (Potentially  Combined): 
Flight  Tests  and  Post-Flight  Evaluation 


•  Flight  test  on  a  test  range 

-  Intended  to  measure  interceptor  performance  in  varied  realistic  conditions 

-  Pre-flight  predictions  are  done  using  constructive  six-degree-of-freedom 
(6-dof)  simulations  (for  test  design  and  range  safety  purposes) 

-  For  flight  test  itself 

■  Interceptor  and  target  missile^are  live  simulation  components 

■  If  interceptor  launch  is  under  operator  control,  the  operator  is  a  live 
simulation  component 

•  Post-flight  evaluation 

-  Intended  to  evaluate  single-  and  multiple-flight  test  performance 

-  Post-flight  “predictions”  (e.g.,  using  actual  wind  conditions)  are  often  done 
using  6-dof  simulations  (for  comparison  to  telemetry  data) 

-  Using  multiple-flight  data,  can  use  data  to  create  better  model  of 
interceptor  guidance  and  control  system  (e.g.,  using  Kalman  filtering 
approach) 
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Example:  Repair  Process  for  a 
Deployed  Military  System  Component 


•  Consider  the  repair  process  for  a  deployed  military  system  component 
(radio)  associated  with  a  communications  van  in  theater 

-  When  the  radio  malfunctions,  what  is  the  initial  repair  process? 

-  If  the  radio  cannot  be  fixed  in  place,  where  does  it  go? 

-  How  many  levels  of  repair  are  implemented? 

-  What  is  the  spare  parts  strategy  and  inventory? 

•  How  would  you  model  the  repair  process  using  a  tool  such  as  Arena? 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Example  Levels/Sequence  of  Repair 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Modeling  the  Repair  Process 


•  Each  possible  point  of  repair  has  a 
probability  that  the  radio  can  be  repaired 
there 


-  If  it  can  be  repaired  there,  there  is  a 
distribution  of  repair  times 


-  If  it  cannot  be  repaired  there: 


There  is  a  distribution  of  times  it 
takes  to  come  to  that  decision 


■  There  is  a  transportation  time  to  the 
next  level  of  repair 


P 


R 


T 


R 


Td 
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Modeling  and  Simulation  in  the  Systems  Engineering  Process 


Modeling  the  Repair  Parts  Supply  Chain 


•  Modeling  Issues 

•  Based  on  reliability  models,  how  many  of 
each  radio  part  should  be  stored  at  each 
repair  point? 

•  When  a  spare  part  is  used  at  a  repair 

point,  from  where  is  a  replacement 
requested?  ' 

•  Based  on  reliability/availability  models  and 
logistics/transportation  cost  issues,  at 
what  spare  parts  inventory  level  at  a  given 
repair  point  should  replacements  be 
shipped? 


Organizational 
Unit  (Org) 


Direct  Support 
Unit  (DSU) 


General  Support 
Unit  (GSU) 


Depot  (Dep) 


Contractor 

(Con) 
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Agenda 


•  MIL-STD-882E  Requirements  for  Software  Safety 

•  DoD  Guidance  for  Software  Safety 

•  Software  System  Safety  Hazard  Analysis 

•  Functional  Hazard  Analysis  (FHA)  for  Software 

•  In-Depth  Safety-Specific  Testing 

•  Requirements  Analysis 

•  Architecture  Analysis 

•  Design  Analysis 

•  Code  Analysis 

•  Wrap  Up 


Learning  Objectives 


Gain  an  understanding  of: 

•  A  framework  for  performing  and  documenting 
MIL-STD-882E-required  software  safety  Level  of 
Rigor  (LoR) 


NOTE:  Blue  font  is  used  in  these  slides  to  highlight  significant  terms 
or  statements. 


Learning  Objectives 


Gain  an  understanding  of : 

•  A  framework  for  performing  and  documenting 
MIL-STD-882E-required  software  safety  Level  of 
Rigor  (LoR) 


NOTE:  This  framework  will  NOT  be  a  detailed  step-by-step  process 
of  exactly  how  to  perform  each  analysis  on  every  system 
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Learning  Objectives 


Gain  an  understanding  of: 

•  A  framework  for  performing  and  documenting 
MIL-STD-882E-required  software  safety  Level  of 
Rigor  (LoR) 

•  How  to  focus  analysis  of  software  requirements 
and  architecture  on  the  command  and  control  of 
Safety-Significant  Functions 


Learning  Objectives 


Gain  an  understanding  of: 

•  A  framework  for  performing  and  documenting 
MIL-STD-882E-required  software  safety  Level  of 
Rigor  (LoR) 

•  How  to  focus  analysis  of  software  requirements 
and  architecture  on  the  command  and  control  of 
Safety-Significant  Functions 

•  How  to  focus  analyses  of  the  design  and  code 
on  Safety-Critical  Decision  Points 


Learning  Objectives 


Gain  an  understanding  of: 

•  A  framework  for  performing  and  documenting 
MIL-STD-882E-required  software  safety  Level  of 
Rigor  (LoR) 

•  How  to  focus  analysis  of  software  requirements 
and  architecture  on  the  command  and  control  of 
Safety-Significant  Functions 

•  How  to  focus  analyses  of  the  design  and  code 
on  Safety-Critical  Decision  Points 

•  How  to  derive  the  safety-specific  test  cases  from 
the  analysis 


MIL-STD-882E  Requirements  for 

Software  Safety 


Some  MIL-STD-882E  Terminology 


Software.  A  combination  of  associated  computer 
instructions  and  computer  data  that  enable  a  computer 
to  perform  computational  or  control  functions.  Software 
includes  computer  programs,  procedures,  rules,  and 
any  associated  documentation  pertaining  to  the 
operation  of  a  computer  system.  Software  includes  new 
development,  complex  programmable  logic  devices 
(firmware),  NDI,  COTS,  GOTS,  re-used,  GFE,  and 
Government-developed  software  used  in  the  system. 


Software  system  safety.  The  application  of  system 
safety  principles  to  software. 


Some  MIL-STD-882E  Terminology 


Software  control  category.  An  assignment  of  the  degree 
of  autonomy,  command  and  control  authority,  and 
redundant  fault  tolerance  of  a  software  function  in 
context  with  its  system  behavior. 

see  Software  eontrol  eategory 

Swei  Software  ehticality  Index 

Level  of  rigor  (LoRf.  A  specification  of  the  depth  and 
breadth  of  software  analysis  and  verification  activities 
necessary  to  provide  a  sufficient  level  of  confidence  that 
a  safety-critical  or  safety-related  software  function  will 
perform  as  required. 


Some  MIL-STD-882E  Terminology 


Safety-critical.  A  term  applied  to  a  condition,  event, 
operation,  process,  or  item  whose  mishap  severity 
consequence  is  either  Catastrophic  or  Critical  (e.g., 
safety-critical  function,  safety-critical  path,  and  safety- 
critical  component). 

Safety-related.  A  term  applied  to  a  condition,  event, 
operation,  process,  or  item  whose  mishap  severity 
consequence  is  either  Marginal  or  Negligible. 

Safety-significant.  A  term  applied  to  a  condition,  event 
operation,  process,  or  item  that  is  identified  as  either 
safety-critical  or  safety-related. 


Some  MIL-STD-882E  Terminology 


Safety-critical  function  (SCF).  A  function  whose  failure 
to  operate  or  incorrect  operation  will  directly  result  in  a 
mishap  of  either  Catastrophic  or  Critical  severity. 

SSF  Safety-Significant  Function 

SSSF  Safety-Significant  Software  Function 
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Requirements  for  Software  Safety 

[from  MIL-STD-882E] 

4.1  General.  When  this  Standard  is  required  in  a 
solicitation  or  contract,  but  no  specific  tasks  are 
included,  only  Sections  3  and  4  apply.  The  definitions  in 

3.2  and  all  of  Section  4  delineate  the  minimum 
mandatory  definitions  and  requirements  for  an 
acceptable  system  safety  effort  for  any  DoD  system. 

■  ■  ■ 

4.3.2  Identify  and  document  hazards.  Hazards  are 
identified  through  a  systematic  analysis  process  that 
includes  system  hardware  and  software,  system 
interfaces  (to  include  human  interfaces) . . . 
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Requirements  for  Software  Safety 


[from  MIL-STD-882E] 

4.4  Software  contribution  to  system  risk.  The 
assessment  of  risk  for  software,  and  consequently 
software-controlled  or  software-intensive  systems, 
cannot  rely  solely  on  the  risk  severity  and  probability. . . 
Therefore,  another  approach  shall  be  used  for  the 
assessment  of  software’s  contributions  to  system  risk 
that  considers  the  potential  risk  severity  and  the  degree 
of  control  that  software  exercises  over  the  hardware. 


Booz  Allen  Hannilton 


14 


Severity  Categories 


Description 

Severity 

Category 

Mishap  Resuit  Criteria 

Catastrophic 

1 

Could  result  in  one  or  more  of  the  following:  death  .  . . 
or  monetary  loss  equal  to  or  exceeding  $1 OM. 

Critical 

2 

Could  result  in  one  or  more  of  the  following:  permanent 
partial  disability,  injuries  or .  .  .  monetary  loss  equal  to 
or  exceeding  $1 M  but  less  than  $1  OM. 

Marginal 

3 

Could  result  in  one  or  more  of  the  following:  injury  .  .  . 
resulting  in  one  or  more  lost  work  day(s) ...  or 
monetary  loss  equal  to  or  exceeding  $1 OOK  but  less 
than  $1M. 

Negligible 

4 

Could  result  in  one  or  more  of  the  following:  injury  or 
occupational  illness  not  resulting  in  a  lost  work  day  . . 
or  monetary  loss  less  than  $1  OOK. 
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Software  Control  Categories 


Name 

Level 

Description 

Autonomous 

(AT) 

1 

Software  functionality  that  exercises  autonomous  control 
authority  .  . .  without  the  possibility  of  predetermined  safe 
detection  and  intervention . . . 

Semi- 

autonomous 

(SAT) 

2 

Software  functionality  that  exercises  control .  .  .  allowing  time 
for  predetermined  safe  detection  and  intervention  by 
independent  safety  mechanisms . . . 

Redundant 
Fault  Tolerant 
(RFT) 

3 

Software  functionality  that  issues  commands  .  . .  requiring  a 
control  entity  to  complete  the  command  function . . . 

Influential 

(INF) 

4 

Software  generates  information  of  a  safety-related  nature 
used  to  make  decisions  by  the  operator . . . 

No  Safety 
Impact 
(NSI) 

5 

Software  functionality  that  does  not  possess  command  or 
control  authority  .  . .  and  does  not  provide  safety-significant 
information . . . 

Software  Safety  Criticality  Matrix 


Severity 

W 

Control 

Catastrophic 

(1) 

Critical 

(2) 

Marginal 

(3) 

Negligible 

(4) 

1  (AT) 

SwCI  1 

SwCI  1 

SwCI  3 

SwCI  4 

2  (SAT) 

SwCI  1 

SwCI  2 

SwCI  3 

SwCI  4 

3  (RFT) 

SwCI  2 

SwCI  3 

SwCI  4 

SwCI  4 

4  (INF) 

SwCI  3 

SwCI  4 

SwCI  4 

SwCI  4 

5  (NSI) 

SwCI  5 

SwCI  5 

SwCI  5 

SwCI  5 

NOTE:  The  Influential  (INF)  SCO  only  applies  to  the  generation  of  ‘safety-related’  information 
for  the  operator. 
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Software  Safety  Levels  of  Rigor 


SwCI 


Level  of  Rigor  Tasks 


SwCI  1 

SwCI  2 

SwCI  3 
SwCI  4 
SwCI  5 


Program  shall  perform  analysis  of  requirements, 
architecture,  design,  and  code;  and  conduct  in-depth 
safety-specific  testing. 

Program  shall  perform  analysis  of  requirements, 
architecture,  and  design;  and  conduct  in-depth  safety- 
specific  testing. 

Program  shall  perform  analysis  of  requirements  and 
architecture;  and  conduct  in-depth  safety-specific  testing. 

Program  shall  conduct  safety-specific  testing. 

Once  assessed  by  safety  engineering  as  Not  Safety,  then 
no  safety  specific  analysis  or  verification  is  required. 
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Safety  Risk  for  Failure  to  Perform  LoR 


RELATIONSHIP  BETWEEN  SwCI,  RISK  LEVEL,  LoR  TASKS,  AND  RISK 

SwCI 

Risk 

Level 

Software  LoR  Tasks  and  Risk  Assessment/Acceptance 

1 

High 

If  SwCI  1  LOR  tasks  are  unspecified  or  incomplete,  the 
contributions  to  system  risk  will  be  documented  as  HIGH  . . . 

2 

Serious 

If  SwCI  2  LOR  tasks  are  unspecified  or  incomplete,  the 
contributions  to  system  risk  will  be  documented  as  SERIOUS  . . . 

3 

Medium 

If  SwCI  3  LOR  tasks  are  unspecified  or  incomplete,  the 
contributions  to  system  risk  will  be  documented  as  MEDIUM  . . . 

4 

Low 

If  SwCI  4  LOR  tasks  are  unspecified  or  incomplete,  the 
contributions  to  system  risk  will  be  documented  as  LOW  . . . 

5 

Not 

Safety 

No  safety-specific  analyses  or  testing  is  required. 
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DoD  Guidance  for  Software  Safety 
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MIL-STD-882E  Guidance  for 
Software  Safety 

from  Tasks  102  and  103  -  System  Safety  Program  Plan  and  Hazard 
Management  Plan] 


1 02/1 03.2.6  Hazard  analysis. 

■  ■  ■ 

i.  Describe  a  systematic  software  system  safety 
approach  to: 

■  ■  ■ 

(4)  Identify  and  assign  the  Software  Criticality 
Index  (SwCI)  for  each  safety-significant  software 
function  (SSSF)  and  its  associated  requirements. 
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MIL-STD-882E  Guidance  for 
Software  Safety 


[from  Task  208  -  Functional  Hazard  Analysis] 

208.2.1  .  .  . 

g.  An  assessment  of  Software  Control  Category 
(SCC)  for  each  Safety-significant  Software  Function 
(SSSF).  Assign  a  Software  Criticality  Index  (SwCI) 
for  each  SSSF  mapped  to  the  software  design 
architecture. 
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JSSEH  Guidance 


4.2.1 .4  Defining  and  Using  the  Software  Criticality 
Matrix 

...  It  is  through  this  prioritization  that  safety-significant 
code  can  receive  the  appropriate  robustness  and  level 
of  rigor  over  the  lifecycle,  while  effectively  managing  the 
critical  resources  of  the  program.  The  most  important 
aspect  of  the  activity  is  that  the  software  with  the 
highest  level  of  control  over  safety-significant  hardware 
must  receive  more  attention  or  level  of  rigor  than 
software  with  less  safety  risk  potential. . . 
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JSSEH  Guidance 

4.2.1 .4  Defining  and  Using  the  Software  Criticality  Matrix 

...  It  is  through  this  prioritization  that  safety-significant 
code  can  receive  the  appropriate  robustness  and  level  of 
rigor  over  the  lifecycle,  while  effectively  managing  the 
critical  resources  of  the  program.  The  most  important 
aspect  of  the  activity  is  that  the  software  with  the  highest 
level  of  control  over  safety-significant  hardware  must 
receive  more  attention  or  level  of  rigor  than  software  with 
less  safety  risk  potential. . .  This  methodology  helps 
prioritize  and  manage  the  critical  resources  of  schedule, 
budget,  and  personnel  associated  with  the  development 
of  the  system. 


JS-SSA  Software  System  Safety: 

Implementation  Process  and  Tasks  Supporting  MIL-STD-882E 


3.5.  [LoR]  Allocations  to  Safety-Significant  Functions 

The  allocation  of  SSFs  to  specific  [LoR]  categories  is 
essential,  both  to  ensure  the  provision  of  rigor  to  the 
functions  of  highest  safety  criticality  and  to  ensure  the 
management  of  the  critical  resources  necessary  to 
implement  that  rigor. . .  [T]he  accomplishment  of  the 
subtasks  ...  must  be  thoroughly  documented  within  the 
artifacts  of  the  safety  analysis. 
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Software  System  Safety 
Hazard  Analysis  -  an  Overview 


Where  to  Focus 


14-5.C. ...  focus  ...  on  hazard  identification  and 
mitigation  of  software  causal  factors,  as  opposed  to 
error  removal. 

14-5.d. ...  focus  ...  on  hazard  and  software  causal 
factor  identification  and  mitigation,  as  opposed  to 
requirements  perfection.  [Software  safety  requirements 
should  be  based  on  mitigating  software  related 
hazards.] 


NAVSEA  SW020-AH-SAF-010,  Section  II,  Weapon  System  Safety  Guidelines  Handbook 
System  Safety  Engineering  and  Management. 
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Software  System  Safety  Hazard  Analysis 


step  1  -  Perform  a  software  Functional  Hazard  Analysis 
(FHA) 
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Software  System  Safety  Hazard  Analysis 


step  1  -  Perform  a  software  Functional  Hazard  Analysis 
(FHA) 

Step  2  -  For  each  SSSF,  perform  (and  document)  all 
required  tasks. 

For  each  analysis  task,  identify: 

a.  Potential  Causal  Factors 

b.  Potential  (or  actual)  Mitigations 

c.  Appropriate  In-Depth  Safety-Specific  Testing  for 
each  CF  and  Mitigation 
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FHA  for  Software 

[the  beginning  of  a  MiL-STD-882E  software  safety  effort] 


Performing  the  Software  FHA 

Step  1  -  Perform  a  software  Functional  Hazard  Analysis 
(FHA) 

a.  Identify  each  Safety-Significant  Function  (SSF)  that 
has  been  allocated  to  software  (a  SSSF). 

b.  Assess  the  level  of  software  control  of  the  function 
(the  Software  Control  Category,  or  SCC). 

c.  Identify  associated  safety  requirements  or  design 
constraints. 

d.  For  highly  critical  (SwCI  1)  SSSFs,  identify  potential 
system  or  software  design  redundancies  to  lower  the 
SwCI  (and  required  LoR). 
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MIL-STD-882E  Guidance  for  FHA 


[from  Task  208  -  Functional  Hazard  Analysis] 


208.1  Purpose  . . .  The  initial  FHA  should  be 
accomplished  as  early  as  possible  in  the  Systems 
Engineering  (SE)  process  to  enable  the  engineer  to  . . 

•  identify  and  document  SCFs,  SCIs,  SRFs,  and 
SRIs; 

•  allocate  and  partition  SCFs  and  SRFs  in  the 
software  design  architecture; 

•  and  identify  requirements  and  constraints  to  the 
design  team. 


A  working  definition  for  ‘Function’ 

The  following  is  a  working  definition  we  will  use  for  the 
term  ’’software  function”  (somewhat  modeled  after  a 
mathematical  function): 

Given  an  input,  or  a  set  of  related  inputs,  a  software 
function  produces  one  or  more  of  the  following 
outcomes: 

•  An  externally  observable  system  action; 

•  Externally  observable  digital  information  that  can 
be  used  by  a  system  operator  or  another  software 
entity;  or 

•  An  internal  change  of  digital  state. 

NOTE:  The  use  of  ‘external’  and  ‘internal’  refers  to  the  context  of  the 
_ software  component(s). _ 
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Naming  a  Function 


Name  each  SSSF  using  a  verb  (performing  an  action)  and 
a  noun  (object  of  the  verb): 

Examples: 

•  Arm  the  warhead 

•  Detonate  the  warhead 

•  Arm  the  booster 

•  Ignite  the  booster 

•  Release  the  missile 

•  Safe  the  booster 

•  Fire  the  weapon 
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Choosing  the  ‘size’  of  a  SSSF 


Use  engineering  judgement  to  choose  the  best  ‘size’  of 
SSSFs  for  effective  and  efficient  analysis  and  test  -  too 
high  a  level  puts  too  much  functionality  all  in  the  same 
“analysis  bucket,”  while  too  low  a  level  breaks  the 
analysis  into  too  many  pieces. 

Some  examples: 

•  Too  high:  Perform  a  Standard  Missile  engagement 

•  Too  low:  Close  the  K1  relay 

•  Good  level:  Arm  the  missile  booster 
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Functional  Failure  Types 

[from  NAVSEA  SW020-AH-SAF-0010] 


Function: 

1  Fails  to  operate 

2  Operates  incorrectly/erroneously 

3  Operates  inadvertently 

4  Operates  at  wrong  time  (early) 

5  Operates  at  wrong  time  (late) 

6  Unable  to  stop  operation 

7  Receives  erroneous  data 

8  Sends  erroneous  data 

9  Conflicting  data  or  information 


Performing  the  Software  FHA 

-  Step  1a-- 

Step  la.  Identify  each  Safety-Significant  Function  (SSF) 
that  has  been  allocated  to  software  (a  SSSF). 

•  Use  the  nine  functional  failure  types  to  reason  about 
the  different  ways  the  SSSF  might  fail  with  potential 
safety  impact. 

Ex.  -  Software-allocated  missile  release  function 
fails  to  operate  after  missile  ignition. 

•  Document  the  level  of  mishap  severity  that  might 
result  from  the  functional  failure. 

Note:  For  the  weapon  systems  and  combat  systems  we  work  with, 
this  is  most  often  CAT  for  software  functional  failures. 
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Performing  the  Software  FHA 

-  Step  1b- 

Step  1  b.  Assess  the  level  of  software  control  of  the 
function  (the  Software  Control  Category,  or  SCC). 

To  claim  Semi-autonomous  SCC,  document  how  each 
SSSF  failure  is  detected  and  what  the  independent 
safety  mechanism  that  mitigates  or  controls  the 
resulting  hazard  is. 

To  claim  Redundant  Fault  Tolerant  SCC,  document 
what  the  redundancies  are  and  how  they  mitigate  or 
control  each  safety-significant  failure  type  for  the 
SSSF. 
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Examples  of  SSSF  Functional  Failures 


•  Safe  weapon  SSSF  fails  to  operate 

•  Arm  warhead  SSSF  operates  inadvertently 

•  Detonate  warhead  SSSF  operates  inadvertently 

•  Detonate  warhead  SSSF  operates  at  wrong  time  (early) 

•  Detonate  warhead  SSSF  operates  at  wrong  time  (late) 

NOTE:  The  SSSF  hazard  severity  or  the  software  control  category  may 
vary  for  each  functional  failure  type. 
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Performing  the  Software  FHA 

-  Step  1c-- 

Step  1c.  Identify  associated  safety  requirements  or  design 
constraints. 

The  safety  requirements  and  design  constraints  are 
mitigations  for  the  safety-significant  SSSF  failures. 
Communicate  these  with  the  system  and  software 
engineers  to  ensure: 

•  They  are  included  in  the  requirements  and  design 
(or  coding  standards)  for  the  system 

•  There  are  appropriate  tests  (or  inspections  or 
analyses)  included  to  validate  the  mitigations  work 
to  control  identified  safety-significant  failures  for 
the  SSSF. 
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Examples  of  Safety  Requirements  and 

Design  Constraints 


•  The  Launcher  shall  include  an  independent  Canister 
Deluge  sub-system  to  command  Canister  Flooding  in 
case  of  Launcher  Overtemperature  or  Missile 
Restrained  Firing. 

•  The  Launcher  shall  only  process  Missile  Launch- 
related  commands  if  the  Launcher  has  been  placed 
in  Tactical  Mode  by  the  Weapon  Control  System. 

•  The  Launcher  shall  allow  the  selection  of  no  more 
than  two  Missiles  for  Launch  at  the  same  time. 
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Performing  the  Software  FHA 

-  Step  1  d  -- 

Step  1d.  For  highly  critical,  SwCI  1  SSSFs,  identify 
potential  system  or  software  design  redundancies  that 
could  lower  the  SwCI  (and  required  LoR). 

These  fault  tolerant  redundancies  are  mitigations  for 
safety-significant  SSSF  failures.  Communicate  these 
with  the  system  and  software  engineers  to  ensure: 

•  They  are  included  in  the  requirements  and  design 
for  the  system 


Ex.  -  The  Boeing  777  primary  flight  software  is  implemented  in  three 
similar  computation  channels  (triple  modular  redundancy),  each  with 
three  dis-similar  ‘computation  lanes’  (written  in  different  programming 
languages). 
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FHA  Advantages 

[from  NAVSEA  SW020-AH-SAF-010  Section  III] 

The  following  are  significant  advantages  of  the  [FHA]: 

a.  Is  easily  and  quickly  performed. 

b. Does  not  require  considerable  expertise. 

c.  Is  relatively  inexpensive,  yet  provides  meaningful 
results. 

d.  Provides  rigor  for  focusing  on  hazards  associated 
with  system  functions. 

e. Good  tool  for  software  safety  analysis. 
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FHA  Disadvantages 

[from  NAVSEA  SW020-AH-SAF-01 0  Section  III] 

The  following  are  disadvantages  of  the  [FHA]: 

a.  ...  it  might  overlook  other  types  of  hazards,  such  as 
those  dealing  with  hazardous  energy  sources  or 
sneak  circuit  paths. 

b.  After  a  functional  hazard  is  identified,  further 
analysis  is  required  to  determine  if  the  causal  factors 
are  possible. 

c.  Cannot  completely  replace  the  need  for  a  PHA. 
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In-Depth  Safety-Specific  Testing 
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In-Depth  Safety-Specific  Testing 


1  .In-Depth  Safety-Specific  Testing  should  be  derived  from 
the  software  safety  analyses 
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In-Depth  Safety-Specific  Testing 


1  .In-Depth  Safety-Specific  Testing  should  be  derived  from 
the  software  safety  analyses 

2. Test  cases  should  be  assigned  to  appropriate  test 
events 
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In-Depth  Safety-Specific  Testing 


1  .In-Depth  Safety-Specific  Testing  should  be  derived  from 
the  software  safety  analyses 

2.  Test  cases  should  be  assigned  to  appropriate  test 
events 

3.  Ensure  results  are  captured  for  safety  evidence 


Booz  Allen  Hannilton 


48 


of  Testing 


[W]e  can  thoroughly  test  hardware  and  get  out 
requirements  and  design  errors  [but  we  c]an  only  test 
a  small  part  of  potential  software  behavior. 

•Leveson,  Nancy  G.,  “A  New  Approach  to  Ensuring  Safety  in  Software  and 
Human  Intensive  Systems.”  SECIE  Safety  in  Software  and  Human  Intensive 
Systems.  July  2009. 


Complacency  may  also  have  been  involved,  i.e.,  the 
common  assumption  that  software  does  not  fail  and 
that  software  testing  is  exhaustive  and  therefore 
additional  software  checking  was  not  needed. 

•Leveson,  Nancy  G.,  “A  Systems-Theoretic  Approach  to  Safety  in  Software- 
Intensive  Systems.”  2004. 
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of  Testing 


[0]ne  of  the  most  important  limitations  of  software 
testing  is  that  testing  can  show  only  the  presence  of 
failures,  not  their  absence.  This  is  a  fundamental, 
theoretical  limitation;  generally  speaking,  the  problem 
of  finding  all  failures  in  a  program  is  undecidable. 

•Paul  Ammonn,  Jeff  Offutt.  Introduction  to  Software  Testing.  2008. 
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of  Testing 


We  cannot  test  software  for  correctness:  Because  of 
the  large  number  of  states  (and  the  lack  of  regularity 
in  its  structure),  the  number  of  states  that  would  have 
to  be  tested  to  assure  that  software  is  correct  is 
preposterous.  Testing  can  show  the  presence  of 
bugs,  but,  except  for  toy  problems,  it  is  not  practical  to 
use  testing  to  show  that  software  is  free  of  design 
errors. 

•David  L.  Parnas,  A.  John  van  Schouwen,  and  Shu  PO  Kwan.  “Evaluation  of 

Safety-Critical  Software.”  Communications  of  the  ACM,  June  1990. 
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An  interview  with  Watts  Humphrey 

(the  “Father  of  Software  Quality”) 

Humphrey:  . .  .  When  you  think  about  a  big  program,  big  complex  system 
program,  2  million  lines  of  code  something  like  that,  and  you  run 
exhaustive  tests,  what  percentage  of  all  the  possibilities  do  you  think 
you’ve  tested?  Any  idea? 

Booch:  Oh  it’s  going  to  be  an  embarrassingly  small  number  probably  in 
the  less  than  20,  30%  would  be  my  guess. .  . 

Humphrey:  You’re  way  off.  Way  off.  I  typically  ask  people  and  I  get  back 
numbers  50%,  30%,  that  kind  of  thing.  I  asked  the  people  at  Microsoft,  the 
Windows  people,  what  they  thought.  And  then  we  chatted  about  it  a  bit 
and  they  said  about  1%. 

Booch:  Oh  my  goodness. 

Humphrey:  And  my  reaction  is  they’re  high  by  several  orders  of 
magnitude. . .  the  number  of  possibilities  is  so  extraordinary  you  literally 
couldn’t  do  a  comprehensive  test  in  the  lifetime  of  the  universe  today. 

“An  Interview  with  Watts  Humphrey,  Part  26:  Catastrophic  Software 
Failures  and  the  Limits  of  Testing”  Watts  S.  Humphrey  and  Grady 
Booch,  Aug  16,  2010,  provided  by  the  Computer  History  Museum. 
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Purpose  of  Testing 


Assess  quality.  This  is  a  tricky  objective  because  quality  is  multi¬ 
dimensional.  . .  For  example,  reliability  is  . . .  about  the  number  of 
reliability-related  failures  that  can  be  expected  in  a  period  of  time  or  a 
period  of  use. .  .To  make  this  prediction,  you  need  a  mathematically  and 
empirically  sound  model  that  links  test  results  to  reliability.  Testing 
involves  gathering  the  data  needed  by  the  model. .  . 

Verify  correctness  of  the  product  It  is  impossible  to  do  this  by  testing. 

Assure  quality.  Despite  the  common  title,  quality  assurance,  you  can’t 
assure  quality  by  testing. .  . 

Assess  conformance  to  specification. . . 

Find  defects. . .  the  classic  objective  of  testing. . .  Generally,  we  look  for 
defects  in  all  interesting  parts  of  the  product. .  . 

Kaner,  C.  “What  Is  a  Good  Test  Case?”  2003. 
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Purpose  of  Safety-Specific  Testing 


In-Depth  Safety-Specific  Testing  should  clearly 
demonstrate  additional  testing  rigor. 

Test  cases  should  attempt  to  show  that: 

1)  Causal  Factor  instances  can  be  realized  an6 

2)  Identified  Mitigations  don’t  work  as  intended 

The  test  scenarios  should  include  credible  “load”  or 
“stress”  relevant  to  the  SSSF. 


Types  of  In-Depth  Testing 

Boundary  limit  testing: 

•  Data  range  limits  (e.g.,  highest  or  lowest  possible  values  of  a 
safety-critical  input,  at  or  near  zero,  or  near/at/over  capacity  limits 
of  a  data  storage). 

•  Timing  limits  (e.g.,  at  the  expiration  of  a  timer  or  time  limit). 

Robustness  testing: 

•  Response  to  abnormal  inputs  and  conditions  while  ensuring  safe 
SSSF  performance,  e.g.,  high  rates  of  new  track  acquisitions  and 
drop-outs. 

Fault  injection  testing: 

•  Response  to  faults  injected  during  SSSF  performance. 

Stress  testing: 

•  Response  to  credible  system  stress  during  SSSF  performance. 
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Types  of  In-Depth  Testing 

Safe  state  transition  testing: 

•  Exercise  all  possible  state  transitions  during  SSSF  performance. 
Out  of  sequence  testing: 

•  Software  response  to  out-of-sequence  inputs  and  conditions 
whiie  ensuring  safe  performance  of  the  SSSF. 

Out-of-range  vaiue  testing: 

•  Assurance  of  safe  performance  of  the  SSSF  in  response  to  out- 
of-range  inputs  or  data  values. 

Error  and  exception  handiing  testing: 

•  Response  to  errors  and  exceptions  during  SSSF  performance. 
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Types  of  In-Depth  Testing 

Timing  anaiysis  testing: 

•  For  safety-critical  hard  reai  time  requirements,  use  targeted 
ioad  or  stress  testing  of  the  time-criticai  SSSF  functionaiity  to 
support  the  findings  of  timing  anaiyses  performed. 

Aigorithm  correctness  testing: 

•  Targeted  stress  testing  of  safety-criticai  aigorithms  associated 
with  the  SSSF. 

Independent  test: 

•  Testing  of  prioritized  SSSFs  by  an  independent  test  team,  if 
determined  to  be  needed  by  analysis. 

Regression  testing: 

•  Focused  regression  testing  of  SwCI  1  or  2  SSSF  as  determined 
from  changes  to  related  functionality. 
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Examples  of  In-Depth  Testing 


•  Script  a  “Restrained  Firing”  in  a  Launcher  followed  immediately 
by  a  communication  failure  and  “hand-ofF’  of  the  Launcher  to  the 
alternate  Launch  Controller: 

-  See  if  all  missile  launches  in  the  Launcher  are  “safed,”  as 
required  after  a  Restrained  Firing 

•  Script  a  second  Launch  Inhibit  Command  just  as  the  first 
Launch  Inhibit  Command  timeout  is  occurring,  which  should 
clear  the  first  Launch  Inhibit  condition 

•  Script  a  “failover”  of  the  primary  Launch  Controller  to  the 
alternate  Launch  Controller  just  after  a  Launch  Inhibit  Command 
has  been  processed. 

•  Script  a  “Restrained  Firing”  during  a  Max  Launch  test  scenario. 
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Requirements  Analysis 
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Safety  Requirements  Analysis  (SRA) 


The  safety  requirements  are  the  driving  force  behind  a 
designer’s  ability  to  design  safety  into  a  system  and  its 
subsystems. . . 

From  a  safety  perspective,  there  are  three  categories  of 
SSRs  [software  safety  requirements] . . .  contributing 
software  safety  requirements  (CSSR),  generic  software 
safety  requirements  [GSSR],  and  mitigating  software 
safety  requirements  (MSSR). 

[from  the  Joint  Software  System  Safety  Engineering  Handbook  (2010)] 
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Generic  Software  Safety  Requirements 

(GSSRs) 


GSSRs  are  requirements  that  have  been  documented 
over  the  years  under  the  heading  of  lessons  learned 
and  best  practices. . .  The  requirements  themselves  are 
not  safety  specific  and  may  not  yet  be  tied  to  a  specific 
system  hazard. 

[from  the  Joint  Software  System  Safety  Engineering  Handbook  (2010)] 
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Some  Example  GSSRs 


The  Launcher  software  shall  adhere  to  all  MISRA 
C++  guidelines  for  safety-critical  software,  with  the 
exception  of  those  documented,  with  rationale  for 
non-compliance,  in  Table  X. 

The  Launcher  software  shall  not  perform  dynamic 
memory  allocation,  except  during  program 
Initialization. 

The  Launcher  software  shall  not  use  C++  templates 
for  any  safety-significant  software  data  objects  or 
functions. 


Contributing  Software  Safety 
Requirements  (CSSRs) 


The  CSSRs  are  requirements  that  should  already  exist 
in  the  specifications  and  were  likely  authored  by 
someone  other  than  a  safety  engineer.  CSSRs  are 
related  to  the  performance  of  the  system  to  accomplish 
its  intended  function  or  mission.  These  requirements 
are  not  present  for  the  mitigation  or  control  of  a  hazard; 
in  fact,  they  will  often  contribute  to  the  existence  of  a 
hazard.  An  example  of  a  CSSR  is  “Fire  the  Weapon.” . . . 

[from  the  Joint  Software  System  Safety  Engineering  Handbook  (2010)] 
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Some  Example  CSSRs 


•  The  Launcher  shall  power  up  the  Missile  for 
preparation  to  launch  on  the  receipt  of  a  valid  Missile 
Select  Command. 

•  The  Launcher  shall  arm  the  Missile’s  First  Stage 
Booster  on  successful  completion  of  Launch 
Preparation. 

•  The  Launcher  shall  apply  Ignition  Power  on  detection 
of  all  Missile-Launcher  Ready  to  Launch  conditions. 

•  The  Missile  shall  arm  the  Warhead  on  detection  of 
Safe  Separation  from  the  Launch  Platform. 
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Mitigating  Software  Safety 
Requirements  (MSSRs) 


MSSRs  are  requirements  derived  from  in-depth  mishap 
and  hazard  causal  analyses. ...  the  safety  engineer 
[performs]  the  safety  analysis  to  determine  whether  the 
GSSRs  have  successfully  mitigated  the  known  causal 
factors  of  the  mishaps  and  hazards. . . 

MSSRs  are  usually  authored  by  safety  engineers,  with 
input  and  assistance  from  the  design  engineers  and 
domain  experts  associated  with  the  design  or 
subsystem  being  analyzed.  These  MSSRs  must  be 
added  to  the  specifications  . . . 

[from  Joint  Software  System  Safety  Engineering  Handbook  (2010)] 
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Some  Example  MSSRs 


•  The  Launcher  Deluge  subsystem  shall  continuously 
monitor  for  Canister  and  Launcher  Overtemperature 
and  for  Restrained  Firing,  and  command  Canister 
Deluge  on  those  Canisters  effected  by  the 
occurrence  of  any  detected  Hazards. 

•  The  Launcher  shall  set  a  75  second  timer  for  the 
completion  of  each  Missile  Launch  Sequence,  and 
Safe  any  selected  Missile  that  has  not  completed  a 
Launch  within  that  time  period. 
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Analysis  of  Requirements 


Assess  all  tagged  CSSRs/MSSRs  for: 

•  Completeness 

•  Potential  conflict  with  other  requirements 


Ambiguity 


Example  Conflicting/Ambiguous 


•  Potential  conflicting  requirements: 

-  Automated  train  doors  must  open  only  when  train 
is  stopped  and  properly  aligned  with  the  platform. 

-  Automated  train  doors  must  open  for  evacuation  in 
the  event  of  an  emergency. 

•  Potential  ambiguous  requirement: 

-  Aircraft  shall  inhibit  thrust  reversal  when  the 
aircraft  is  in  flight. 
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Architecture  Analysis 
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Some  Terminology 

(from  the  JSSSEH  and  other  sources) 

Architecture:  The  organizational  structure  of  a  system 
or  component  (IEEE  61 0.1 2  -  1 990). 

-  ‘Architecture  is  concerned  with  the  selection  of  architectural 
elements,  their  interaction,  and  the  constraints  on  those 
elements  and  their  Interactions’  (Perry  &  Wolf,  1992,  p.  40- 
52). 

-  ‘Architecture  focuses  on  the  externally  visible  properties  of 
software  “components”  (Bass,  Clements,  &  Kazman,  1998). 

System  Architecture:  The  arrangement  of  elements 
and  subsystems  and  the  allocation  of  functions  to  meet 
system  requirements  {INCOSE  Systems  Engineering 
Handbook) . 
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Safety  in  a  Control  System 


In  control  theory,  open  systems  are  viewed  as  interrelated 
components  that  are  kept  in  a  state  of  dynamic 
equilibrium  by  feedback  loops  of  information  and  control. 

. . .  [A]ccidents  often  occur ...  as  a  result: 

1 .  Incorrect  or  unsafe  control  commands  are  given 

2.  Required  control  actions  (for  safety)  are  not  provided 

3.  Potentially  correct  control  commands  are  provided  at 
the  wrong  time  (too  early  or  too  late),  or 

4.  Control  is  stopped  too  soon  or  applied  too  long. 

Nancy  G.  Leveson,  Engineering  a  Safer  World:  Systems  Thinking  Applied  to  Safety,  201 1 . 
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The  Classic  “Control  Loop” 


Control  Algorithms 
Set  Points 


Disturbances 
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Inter-Process  Architecture  Analysis 


•  Treat  each  distributed  SSSF  as  a  control  loop  allocated 
across  the  system  architecture. 

•  Think  of  ways  the  control  or  feedback  signals 
(messages)  might  be  corrupted,  delayed  or  lost 
(potential  Causal  Factors). 

•  For  each  of  the  Causal  Factors  identified,  think  of 
existing  or  potential  mitigations. 
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System  Path/Thread  Analysis 
for  a  ‘Safe  Weapon’  SSSF 


[] 

Safe  Wpn  ^ 

[] 

[] 

[] 

[] 

Safe  Wpn  ^ 

[] 

[] 

[] 

^Ack/Nak 

[] 

[] 

^WILCO  (or 

[] 

[] 

[] 

CANTPRO) 

[] 

[] 

CSCI  =  Computer  Software  Configuration  Item 
WILCO  =  “Will  Comply” 

CANTPRO  =  “Cannot  Process” 

Ack  =  ‘Valid’  Message  Acknowledge 

Nak  =  ‘Invalid’  Message  (Negative)  Acknowledge 
Safe  Wpn  =  Safe  Weapon 
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More  Robust  ‘Architecture’ 
for  a  ‘Safe  Weapon’  SSSF 


[] 

Safe  Wpn  ^ 

□ 

[] 

[] 

^Ack/Nak 

n 

[] 

[] 

n 

Safe  Wpn*  A 

[] 

[] 

□ 

^Ack/Nak 

[] 

[] 

□ 

^HAVCO** 

[] 

[] 

^HAVCO** 

n 

[] 

[] 

*  CSCI  1  timer  on 
CSCI  2’s 

HAVCO/CANTCO 

response 

□ 

**  or  CANTCO 

[] 

CSCI  =  Computer  Software  Configuration  Item 
HAVCO  =  “Have  Complied” 

CANTCO  =  “Cannot  Comply” 

Ack  =  ‘Valid’  Message  Acknowledge 

Nak  =  ‘Invalid’  Message  (Negative)  Acknowledge 
Safe  Wpn  =  Safe  Weapon 
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System-Theoretic  Accident  Modei  and 

Processes  (STAMP) 


In  systems  theory,  emergent  properties,  such  as  safety,  arise  from 
the  interactions  among  the  system  components.  The  emergent 
properties  are  controlied  by  imposing  constraints  on  the  behavior 
and  the  interactions  among  the  components.  Safety  then  becomes 
a  controi  probiem  where  the  goal  of  the  control  is  to  enforce  the 
system  constraints.  Accidents  resuit  from  inadequate  control  or 
enforcement  of  safety-related  constraints  on  the  deveiopment, 
design,  and  operation  of  the  system. 

. . .  Feedback  is  a  basic  part ...  of  treating  safety  as  a  control 
problem.  Information  flow  is  a  key  in  maintaining  safety. 


Nancy  G.  Leveson,  Engineering  a  Safer  World:  Systems  Thinking  Applied  to  Safety,  201 1 . 


STAMP  View  of  System  Safety 


Inadequate  Enforcement 
of  Safety  Constraints  on 
Process  Behavior 


Inadequate  Control 


From  Figure  4-7  of  Nancy  G.  Leveson, 
Engineering  a  Safer  World:  Systems 
Thinking  Applied  to  Safety,  2011 


Hazardous 

Process 


Hierarchical  Safety  Control  Structure 


Hazardous  System  State 
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General  Control  Loop  with  Causal  Factors 

(from  Safety  Assurance  in  NextGen,  NASA/CR-201 2-21 7553) 


Provided  Control 
Action 

(inappropriate, 
inetfective,  or 
missing) 


Control  input  or  external 
information  wrong  or  missing 

Controller  1 


Actuator , , 


Control  Algorithm 

Process  Model 

(Flaws  in  creation. 

(Inconsistent, 

process  changes 

incomplete,  or 

incorrect  adaptation 

incorrect) 

or  modification) 

Inadequate 

operation 


Controller  2 


Sensor 


Received  Feedback 
(Inadequate,  missing, 
or  delayed) 


Received  Control 
Action  (delayed, 

etc.)  Controlled  Process 


Inadequate 

operation 


Conflicting  control  actions 


Process  input  missing 
or  wrong 


Component  failures 
Changes  over  time 


Provided  Feedback 
(Incorrect,  no 
information  provided, 
measurement 
inaccuracies,  delays) 


Unidentified  or  out-of- 
range  disturbance 


Process  output  contributes 
to  system  hazard 

STAMP/STPA  provides  a  good  framework 
for  analyzing  SSSF  safety  control. 
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Some  Thoughts  On  STAMP,  STPA,  and 
Meeting  MIL-STD-882E  Required  LoR 

We  do  not  assign  a  SwCI  because  in  STAMP  software  can  and 
should  be  treated  in  the  same  way  as  hardware,  i.e.,  the  hazards  are 
identified  along  with  causal  scenarios  leading  to  the  hazards.  Then 
engineers  can  eliminate  or  mitigate  those  causes  according  to 
standard  system  safety  practice  and  design  precedence  . . . 

Nancy  G.  Leveson,  “STPA  (System-Theoretic  Process  Analysis)  Compliance  with  Army  Safety 
Standards  and  Comparison  with  SAE  ARP  4761 a  whitepaper  on  the  compliance  of  STPA  with 
MIL-STD-882E  and  Army  AMCOM  Regulation  385-17. 


Booz  Allen  Hannilton 


79 


Some  Thoughts  On  STAMP,  STPA,  and 
Meeting  MIL-STD-882E  Required  LoR 

We  do  not  assign  a  SwCI  because  in  STAMP  software  can  and 
should  be  treated  in  the  same  way  as  hardware,  i.e.,  the  hazards  are 
identified  along  with  causal  scenarios  leading  to  the  hazards.  Then 
engineers  can  eliminate  or  mitigate  those  causes  according  to 
standard  system  safety  practice  and  design  precedence  . . . 

Nancy  G.  Leveson,  “STPA  (Systenn-Theoretic  Process  Analysis)  Compliance  with  Army  Safety 
Standards  and  Comparison  with  SAE  ARP  4761 a  whitepaper  on  the  compliance  of  STPA  with 
MIL-STD-882E  and  Army  AMCOM  Regulation  385-17. 

THOUGHTS: 

•  STAMP/STPA  is  a  very  good  framework  for  software  safety 
architecture  analysis. 
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Some  Thoughts  On  STAMP,  STPA,  and 
Meeting  MIL-STD-882E  Required  LoR 

We  do  not  assign  a  SwCI  because  in  STAMP  software  can  and 
should  be  treated  in  the  same  way  as  hardware,  i.e.,  the  hazards  are 
identified  along  with  causal  scenarios  leading  to  the  hazards.  Then 
engineers  can  eliminate  or  mitigate  those  causes  according  to 
standard  system  safety  practice  and  design  precedence  . . . 

Nancy  G.  Leveson,  “STPA  (Systenn-Theoretic  Process  Analysis)  Compliance  with  Army  Safety 
Standards  and  Comparison  with  SAE  ARP  4761 a  whitepaper  on  the  compliance  of  STPA  with 
MIL-STD-882E  and  Army  AMCOM  Regulation  385-17. 

THOUGHTS: 

•  STAMP/STPA  is  a  very  good  framework  for  software  safety 
architecture  analysis. 

•  It  would  be  a  very  “heavy  lift”  for  an  individual  program  or  PFS  to 
make  the  case  that  STAMP/STPA  is  replacement  for  required  LoR. 
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Some  Thoughts  On  STAMP,  STPA,  and 
Meeting  MIL-STD-882E  Required  LoR 

We  do  not  assign  a  SwCI  because  in  STAMP  software  can  and 
should  be  treated  in  the  same  way  as  hardware,  i.e.,  the  hazards  are 
identified  along  with  causal  scenarios  leading  to  the  hazards.  Then 
engineers  can  eliminate  or  mitigate  those  causes  according  to 
standard  system  safety  practice  and  design  precedence  . . . 

Nancy  G.  Leveson,  “STPA  (Systenn-Theoretic  Process  Analysis)  Compliance  with  Army  Safety 
Standards  and  Comparison  with  SAE  ARP  4761 a  whitepaper  on  the  compliance  of  STPA  with 
MIL-STD-882E  and  Army  AMCOM  Regulation  385-17. 

THOUGHTS: 

•  STAMP/STPA  is  a  very  good  framework  for  software  safety 
architecture  analysis. 

•  It  would  be  a  very  “heavy  lift”  for  an  individual  program  or  PFS  to 
make  the  case  that  STAMP/STPA  is  replacement  for  required  LoR. 

•  My  experience  has  been  that  MANY  software  problems  are  not  at 
the  architecture  level  (and  can’t  be  eliminated  there). 
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Design  Analysis 
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What  is  “Design”? 


‘Design  focuses  on  the  properties  of  software 
“components”  that  are  not  externally  visible.’ 

[S.  Whitford,2015] 
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Design 

What  is  NOT  Externally  Visible 

What  is  NOT  externally  visible? 

-  The  organization  of  elements  inside  each  software  component, 
e.g.: 

o  Is  it  object  oriented  (Java,  C++)  or  not  (C,  Assembier)? 
o  Is  it  singie  threaded  or  muiti-threaded? 

-  The  data  flow  between  the  elements  inside  each  software 
component,  e.g.: 

o  Message  passing 
o  Caii  parameters 
o  Giobai  data 

-  The  controi  flow  between  the  elements  inside  each  software 
component,  e.g: 

o  Procedure/function  calls 

o  Semaphores/mutexes/monitors 
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Safety-Critical  Decision  Points 


Most  SwCI  1  or  SwCI  2  SSSFs  are  safety-critical  because  the 
software  has  command  authority  over  a  safety-critical  system 
action. 


Safety-Critical  Decision  Points 


•  Most  SwCI  1  or  SwCI  2  SSSFs  are  safety-critical  because  the 
software  has  command  authority  over  a  safety-critical  system 
action. 

•  The  software  is  therefore  responsible  for  making  the  decision  to 
take  that  action,  often  the  release  of  lethal  energy. 
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Safety-Critical  Decision  Points 


•  Most  SwCI  1  or  SwCI  2  SSSFs  are  safety-critical  because  the 
software  has  command  authority  over  a  safety-critical  system 
action. 

•  The  software  is  therefore  responsible  for  making  the  decision  to 
take  that  action,  often  the  release  of  lethal  energy. 

•  If  the  data  used  to  make  the  safety-critical  decision  is  corrupted 
or  stale,  the  software  can  make  the  wrong  decision  with 
catastrophic  results. 
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Safety-Critical  Decision  Points 


•  Most  SwCI  1  or  SwCI  2  SSSFs  are  safety-critical  because  the 
software  has  command  authority  over  a  safety-critical  system 
action. 

•  The  software  is  therefore  responsible  for  making  the  decision  to 
take  that  action,  often  the  release  of  lethal  energy. 

•  If  the  data  used  to  make  the  safety-critical  decision  is  corrupted 
or  stale,  the  software  can  make  the  wrong  decision  with 
catastrophic  results. 

•  Design  (and  Code)  Analysis  should  be  focused  on  how  the 
software  maintains,  or  could  fail  to  maintain,  the  integrity  of  the 
data  used  at  each  Safety-Critical  Decision  Point  in  the  SSSF. 
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SCDP:  An  Example 


Is  it  safe  to  launch  the  missile? 

-  Was  a  valid  Launch  Command  received  from  the  Operator? 

-  Is  the  Cell  Hatch  fully  open? 

•  Does  the  Cell  Hatch  No.  1  sensor  report  “open”? 

•  Does  the  Cell  Hatch  No.  2  sensor  report  “open”? 

-  Is  the  Uptake  Hatch  fully  open? 

•  Does  the  Uptake  Hatch  No.  1  sensor  report  “open”? 

•  Does  the  Uptake  Hatch  No.  2  sensor  report  “open”? 

-  Has  it  been  long  enough  since  the  last  missile  lunched? 

-  Is  the  Close-In  Weapon  System  (CIWS)  nof  currently  firing? 
(Implemented  as  a  launchinhibited  Boolean  (TRUE/FALSE) 
data  Item.) 
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‘Launch  Inhibited’  impiemented 
with  multiple  threads 


Thread  A: 


[  Launch  Missile  Command  received  ] 


boolean  isMsILaunchOK  () 

If . . .  hatch  statuses  and 

last  missile  launch  time  are  “ok” 
. .  .  (launchinhibited  ==  FALSE) 
return  TRUE 
else 

return  FALSE 


launchinhibited  is  set  to  TRUE  when  a  CIWS  engagement  is  about  to  start. 
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‘Launch  Inhibited’  impiemented 
with  muitipie  threads  (cont’d) 


/^read  B  (higher  priority): 

/^^read  C  (lower  priority): 

[  Launch  Inhibit  Command 

[  20s  Launch  Inhibit  timer 

received  ] 

expires  ] 

setLaunchInhibit  () 

clearLaunchInhibit  () 

...  if  old  timer  active,  cancel  it 

. .  .  launchinhibited  =  FALSE 

.  .  .  launchinhibited  =  TRUE 

.  . .  Initiate  a  20s  timer  to  clear 

inhibit 

J 

L  J 

Intent  is  to  clear  a  pre-existing  Launch  Inhibit  condition  after  20  seconds. 
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Analysis  of  ‘Launch  Inhibited’ 


Thread  B  (higher  priority): 


[  Receipt  of  new  Launch 
Inhibit  command  unblocks 
thread  ] 


setLaunchInhibit  () 

...  if  old  timer  active,  cancel  it 
(but,  it’s  too  late) 

.  .  .  launchinhibited  =  TRUE 
.  . .  Initiate  a  (new)  20s  timer 
[thread  blocks  on  task 
completion] 


Thread  C  (lower  priority): 


[  Old  20s  Launch  Inhibit 
timer  expires  ] 


clearLaunchInhibit  () 

.  launchinhibited  =  FALSE 


Timer  intended  to  clear  OLD  Launch  Inhibit  condition  clears  NEW  one  instead!. 
A  data  synchronization  mechanism  should  be  used  to  protect  the  shared  data  item. 
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Some  Sources  of 
Design  Causal  Factors 

Establish  the  pros  and  cons  of  the  design  of  each 
software  component  to  which  the  SSSF  is  allocated 
and  determine  whether  they  could  be  Causal  Factors  or 
Mitigations  for  a  SSSF  functional  failure  due  to  an 
erroneous  Safety-Critical  Decision  by  the  software.  (It’s 
all  about  the  safety-critical  data  integrity.) 
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Some  Sources  of 
Design  Causal  Factors 

Establish  the  pros  and  cons  of  the  design  of  each 
software  component  to  which  the  SSSF  is  allocated 
and  determine  whether  they  could  be  Causal  Factors  or 
Mitigations  for  a  SSSF  functional  failure  due  to  an 
erroneous  Safety-Critical  Decision  by  the  software.  (It’s 
all  about  the  safety-critical  data  integrity.) 

Design  weaknesses  with  respect  to  data  integrity,  e.g.: 

•  Shared  data  “race  conditions” 

•  Loss  of  data  in  software  “failovers” 

•  Failure  to  refresh  temporal  data 

•  Unhandled  exceptions 
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Multi-(two)threaded  Design 


Variables  or  objects  in  the  heap  or  data  can  be  shared  by  the 
threads.  This  can  lead  to  race  conditions  or  thread  deadlock. 
(Text can  also  be  shared,  but  (usually)  does  not  change  in 
value.) 
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Pros  and  Cons  of 
Multi-threaded  Design 

Pros  for  multi-threaded  design: 

•  Allows  software  to  be  more  responsive  to  an  unpredictable 
external  environment  (new  inputs  from  an  operator,  another 
computer,  or  a  sensor) 

•  Each  thread  can  be  ‘appropriately  prioritized’ 

Cons  for  single  threaded  design: 

•  Improperly  synchronized  threads  can  corrupt  shared  data 

•  Improperly  synchronized  threads  can  deadlock  (block  each 
other  forever) 

•  Improperly  prioritized  threads  can  cause  starvation  or 
unpredictable  delays 

•  Much  more  difficult  to  analyze  or  test  than  single-threaded 
designs 
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On  the 


with  Multi-threading 


‘Concurrency  in  software  is  difficult.  However,  much  of 
this  difficulty  is  a  consequence  of  the  abstractions  for 
concurrency  that  we  have  chosen  to  use.  The  dominant 
one  in  use  today  for  general-purpose  computing  is 
threads.  But  non-trivial  multi-threaded  programs  are 
incomprehensible  to  humans.’ 


[The  Problem  with  Threads,  Technical  Report  No.  UCB/EECS-2006-1,  Edward  A. 
Lee,  Professor,  Chair  of  EE,  Associate  Chair  of  EECS,  University  of  California  at 
Berkley,  January  10,  2006] 
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Single  Threaded  Design 

A  (usually  infinite)  loop,  for  an  embedded  program  to  “do  its  thing 
forever.” 

-  Checks  for  input(s)  [e.g.,  messages,  sensor  inputs] 

-  Performs  any  necessary  processing  of  the  input(s) 

-  Produces  output(s)  [e.g.,  messages,  actuator  control  signals] 

Example: 
int  main(void) 

{  //  initialization  code  here  -  done  once 
for  ( ;  ; )  //  or  while  (true)  or  while  (1 ) 

{ //  read  or  detect  stuff 
//do  some  calculation 
//  write  or  command  stuff 

} 


Pros  and  Cons  of 
Single  Threaded  Design 


Pros  for  single  threaded  design: 

•  Easier  to  perform  analysis  (e.g.,  design,  code,  worst  case 
timing) 

•  Easier  to  implement  the  first  time 

Cons  for  single  threaded  design: 

•  Delay  in  responding  to  external  inputs 

•  Can  become  a  bottleneck  in  the  larger  system 

•  Hard  to  prioritize  multiple  competing  “tasks” 

•  Must  implement  the  details  for  handling  all  I/O 

•  Becomes  hard  to  maintain  as  more  functionality  is  added 
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Single  Threaded  Design 
With  Interrupt  Service 


•  With  few  exceptions,  Interrupt  Service  Routines  (ISRs)  should  be  short 
and  sweet.  For  input,  read  the  data  into  a  buffer  or  queue,  set  a  flag  for 
‘main’  to  see,  then  get  out  of  the  way  (let  ‘main’  process  the  data). 

•  Non-atomic  access  by  ‘main’  to  data  shared  with  an  ISR  must  be 
protected  from  potential  corruption  (e.g.,  locking  out  the  interrupt  that 
drives  the  ISR  while  ‘main’  is  reading  from  or  writing  to  the  shared  data). 
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Pros  and  Cons  of 
Design  With  interrupt  Service 


Pros  for  single  threaded  design  with  interrupt  service: 

•  Somewhat  more  responsive  to  external  inputs 

•  Relatively  easy  to  perform  analysis  (e.g.,  design,  code,  worst 
case  timing) 

•  Still  easy  to  implement  the  first  time 

Cons  for  single  threaded  design  with  interrupt  service: 

•  Delay  in  responding  to  external  inputs 

•  Main  loop  can  still  become  a  bottleneck  (input  queue 
overflow,  delay  in  responding  to  external  system) 

•  Still  hard  to  prioritize  multiple  competing  “tasks” 

•  Potential  for  corrupting  data  shared  between  ISRs  and  ‘main’ 

•  Still  hard  to  maintain  as  more  functionality  is  added 


Booz  Allen  Hannilton 


102 


Details  of  Pulse  Width  Modulation 
(Mis-)handled  in  the  ISR 


•  A  Programmable  Power  Supply  was  implemented  so  that  almost  all 
processing  of  sensors  and  control  commands  for  pulse-width 
modulation  (PWM)  of  the  power  output  to  power  up  missiles  in  a 
launcher  for  preparation  to  launch  was  performed  inside  the  ISR. 


•  When  a  new  missile  was  introduced,  the  interrupt  occurred  every  1 0  u- 
sec’s  and  the  ISR  to  11  u-sec’s  to  execute  the  additional  processing  for 
the  power  requirements  for  the  new  missile’s  launch  preparation. 
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An  ISR  /  ’main’  example 
of  non-atomic  data  sharing 


•  Non-atomic  access  by  ‘main’  to  data  shared  with  an  ISR  must  be 
protected  from  potential  corruption  (e.g.,  locking  out  the  interrupt  that 
drives  the  ISR  while  ‘main’  is  reading  from  or  writing  to  the  shared  data). 

•  Inertial  Measurements  include  several  values  -  linear  accelerations  (x, 
y,  and  z)  and  rotational  measurements  (about  each  axis),  is  the  IMU 
ISR  locked  out  while  ‘main’  is  reading  the  shared  IMU  data? 
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Code  Analysis 
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Code  Analysis  vs.  Design  Analysis 


The  difficulty  of  using  the  term  "design"  in  relation  to 
software  is  that  in  some  sense,  the  source  code  of  a 
program  is  the  design  for  the  program  that  it  produces. 


[l/V'/Tc/pecZ/a  article  on  “Software  Design,”  February  7,  2015] 
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Focus  for  LoR  1  Code  Analysis 


SwCI  1  code  is  typically  responsible  for  releasing  potentially 
catastrophic  energy  or  for  detecting  a  potentially  catastrophic 
hazardous  condition.  Either  way  that  usually  involves  one  or  more 
Safety-Critical  Decision  Points  (SCDPs)  in  the  software.  These 
SCDPs  use  one  or  more  software  data  items  to  make  the  decision. 

•  Focus  code  analysis  on  identification  of  internal  data  items 
used  by  software  to  make  critical  decisions  to  perform  a 
safety-critical  action  or  not. 

o  Scope  may  expand  as  analysis  progresses. 

•  Investigate  how  a  data  item’s  value  is  set  and  referenced  by 
the  software. 

•  Static  or  dynamic  code  analysis  tools  should  be  used  for  a 
detailed  analysis  and  to  document  important  technical 
aspects. 


Program  Slicing 


In  computer  programming,  program  slicing  is  the 
computation  of  the  set  of  programs  statements,  the 
program  slice,  that  may  affect  the  values  at  some 
point  of  interest,  referred  to  as  a  slicing  criterion. 
Program  slicing  can  be  used  in  debugging  to  locate 
source  of  errors  more  easily.  Other  applications  of 
slicing  include  software  maintenance,  optimization, 
program  analysis,  and  information  flow  control. 

[Wikipedia ari\c\e  on  “Program  Slicing,”  March  17,  2015] 
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Some  Code  Analysis  Tools 

Tools  to  help  an  analyst  explore  the  code: 
-Eclipse  (Java,  C/C++)  (Open  Source  Software) 
-NetBeans  (Java,  C++) 

-Understand  for  C++/Java  (SCI  Tools) 

Tools  to  do  automated  static  code  analysis: 
-CodeSonar  (GrammaTech) 

-Klocwork  (Rogue  Wave) 

-Code  Advisor  (Coverity) 

-PC-lint  (Gimpel  Software) 


Code  Analysis 


1  .For  each  SwCI  1  SSSF,  identify  and  locate  the  SCDPs 
associated  the  SSSF. 

2.  Using  appropriate  automated  or  semi-automated  code 
analysis  tools,  perform  a  “backward  flow”  analysis  of 
the  code  from  safety-critical  decision  points  in  the 
software. 

3.  Based  on  the  results  of  the  Requirements,  Architecture, 
and  Design  Analyses,  perform  other  appropriate  code 
analyses,  especially  analysis  of  the  implementation  of 
safety  critical  mitigations  for  the  SSSF. 
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Code  Analysis 

-  step  1  “ 

1 .  For  each  SwCI  1  SSSF,  identify  and  locate  the 
SCDPs  associated  the  SSSF. 

•  Locate  the  code  that  performs  energy  release,  e.g., 
weapon  firing,  detonation,  booster  ignition,  (potential 
Causal  Factor)  or  that  detects  and  responds  to  a 
hazardous  condition. 
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An  Example  L-DETS  SCDP 


Source  code  for  safety-critical  function  SetFirePulse  and  associated  functions  in  file  SafetyCritical_RX.c 


II 

H  @Function  void  SetFirePulse(void) 

// . 


//  @Description  This  function  applies  a  30  millisecond  firing  Pulse  to  detonate  the  unit. 

void  SetFirePulse(vo^d) 

{  /' 


Is  detonation  currently  enabled? 


(  if(G_SCV.SC_DisableSafetyCriticalProcessing  ==  SC_PROCESSING_ENABLED)  \ 


if(IsAmiPinRemoved()  =  ARM_PIN_HAS_BEEN_REMOVED) 


{ 


^  //  When  pin  is  pulled  we. get adrigh 

FIREPULSEIPORT  =  1; - 

G_SCV_PortFImage  |=  FIRE  PULSE  BIT; 
G_SCV.SC_DetonatorHasFired  =  DETON  ATOR_HAS_FIRED; 
DelayMilliSecondsNoInterrupt(30); 

SetToSafeStateO; 
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Code  Analysis 

-  step  2  - 

2.  Using  appropriate  automated  or  semi-automated 
code  analysis  tools,  perform  a  “backward  flow”  analysis 
of  the  code  from  safety-critical  decision  points  in  the 
software. 

•  The  analysis  should  focus  on  identifying  potential 
causes  of  stale  or  corrupt  data  being  present  at  the 
safety-critical  decision  point. 
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An  Example  L-DETS  SCDP  (cont’d) 


Control  Flow  Analysis:  How  is  SetFirePulse  called  in  the  L-DETS  Detonator  software? 

Callers  of  SetFirePulseQ  -  /LDETS/5rG/SafetyCritical RX.c  -  in  workspace 

j  SetFirePulseQ  :  void 

j  DetonateUnitO  :  void 

j  PerforinConninand[const  SC_Comnnand_Message_u  :  int 
j  ReceiveMessageO  :  int 
j  ListenModeO  :  void 
•  mainO :  int 

j  ReceiveMessageQ  :  int 
j  ListenModeQ  :  void 
*  niainO :  int 

SetFirePulse  is  only  called  from  the  function  DetonateUnit,  which  is  called  on  two 
paths  within  the  “main”  thread:  one  if  the  Fire  Command  is  received  directly  from 
the  Controller  and  the  second  if  it  has  been  forwarded  from  another  Detonator. 
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An  Example  L-DETS  SCDP  (cont’d) 


Data  Flow  Analysis:  Where/how  is  SC_DisableSafetyCriticalProcessing  updateded? 

Rfencss  to  '(anQnymQU5]::SC Disabl£SafetyCriticalPrQC£ssing'  (11  matches] 


j  0  LDETS 
J  sr[ 

j  @  SafetyCritical_RX,c  i|ll  matches] 


SC_DisableSafetyCriticalProcessing  is  only  enabled  and  disabled  at 
five  locations  in  the  software.  Understanding  the  purpose  and  use  of 
each  location  is  needed  to  assess  for  potential  weaknesses  or  problems.. 


•  ChargeCapacitor,  Iine82:  if[G_SCV,SC_DisableSafetyCriticalProcessing  ==  SC.PROCESSINGJNABLED] 

•  EnableFiring,  line  109:  if[G_SCV,SC_Di^eSafetyMcaJ^oc_^^  ^PRO£ESSING_ENABLED]  /  / 

•  SetFirePulsiafi&i?5:1f(C  OTOisableS^^^^^  ==  SC.PROCESfflTG.MBrffi]—  ^  ^ 

^  I  eSaf  etyC  riti  c  a  I P  ro  c  es  s  i  n  g,  I  i  n  e  336 :  G_SC  V,SC_D  i  s  a  b  I  eSafetyC  riti  c  a  I P  ro  c  es  s  i  ng  =  SC_P  RO  C  E  SSIN  G_E  NAB  LED;  '  ^ 

/ *  *  DisableSafetyCriticalProcessing,  line 354:  G_SCV,SC_Disable5afetyCriticalProcessing  =  SC_PROCE55ING_DMBLED;  \  \ 

C lea rSafetyCriticalVariables, line 377:  G_SCV,SC_DisableSafetyCriticalProcessing  =  SC_PR0CE5SING_ENABLED; 

•  l^ar^aieljiCnticalVariables,  line  381;  G_5CV,SC_DisableSafetyCriti  cal  Pro  cessing  =  SC_PROCE5SING_DISME0>  ^  ! 

t  Clea^SafetyC^it«a^^^^ti^le4el:^4JSil:M,Sr_nisabk^^^^^  / 

•  ClearSafetyCrMV£i^bMn^6'jfLGjg,OisableS^^^^^  S^ROCE^.^ABLED  ]  ; 

^  f^QwerUpCIearSafetyCriticalVariables,  Iir64i0;  G_SCV,SC_Disabl£SafetyCriticalProc£Siingi  =  ^Pm5aNGJSAJliD;:i:^ 

•  PowerUpCIearSafetyCriticalVariables,  Iine468;  if[G_SCV,SC_Disable5afetyCriticalProcessing  1=  5C_PROCE55ING_DMBLED] 

Blue  highlighting  indicates  SC_DisableSafetyCriticalProcessing  is  referenced  but  not  changed. 
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An  Example  L-DETS  SCDP  (cont’d) 


Data  Flow  Analysis:  Where/how  is  SC_DisableSafetyCriticalProcessing  updateded? 

Rfencss  to  '(anQnymQU5]::SC Disabl£SafetyCriticalPrQC£ssing'  (11  matches] 


SC  DisableSafetyCriticalProcessing  is  only  enabled  at  two  locations  in 
the  software.  ' 


A  &  LDETS 

J  sr[ 

A  @  SafetyCritical_RX,c  i|ll  matches] 

•  ChargeCapacitor,  Iine82:  if[G_SCV,SC_DisableSafetyCriticalProcessing  ==  SC.PROCESSINGJNABLED] 

•  EnableFiringJinel09:  if[G_SCV,SC_DisableSafetyCriticalProcessing  ==  SC_PROCESSING_ENABLED] 

•  SetFireP^^  linel]5|_if£_mC^bJ^fet/CritMP^^  _ ^  ; 

"^EnableSafetyCriticalProcessing,  line  336;  G_SCV,SC_DisableSafetyCriti  cal  Processing:  SC_PRQCESSING_ENABLED: 

•  DisableSafetyCriticalProcessinq,  line 354:  G  SCV,SC  DisableSafetyCriticalProcessinq :  SC_PROCESSING_DMBLED;  ; 

I  ea  rSaf  ety  C  riti  c  a  I  Va  ri  a  b  I  es,  Iine377:  G_SC  V,SC_D  i  s  a  b  I  eSafetyC  riti  c  a  I P  ro  c  es  s  i  ng  =  SC_P  RO  C  E  55IN  G_E  NAB  LED; _ 

•  ClearSafetyCriticalVariables,  Iine381:  G_SCV,SC_DisableSafetyCriticalProcessing!=  SC_PR0CE5SING_DMBLED; 

•  ClearSafetyCriticalVariables,  Iine401;  if[G_SCV,SC_Disable5afetyCriticalProcessing  1=  SC_PROCE55ING_ENABLED) 

•  ClearSafetyCriticalVariables,  Iine416;  if[G_SCV,SC_DisableSafetyCriticalProcessing  1=  SC_PR0CE5SING_DMBLED) 

•  PowerUpCIearSafetyCriticalVariables,  Iine450:  G_SCV,SC_DisableSafetyCriticalProcessing=  SC_PROCESSING_DMBLED; 

•  Po  werU  pC  lea  rSafety  Critical  Variables,  Iine468;  if[G_SCV,SC_Disable5afetyCriticalProcessing  1=  SC_PROCE55ING_DMBLED] 

Blue  highlighting  indicates  SC_DisableSafetyCriticalProcessing  is  referenced  but  not  changed 
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Code  Analysis 

-  Step  3  -- 

3.  Based  on  the  results  Requirements  Analysis,  Architecture 
Analysis,  or  Design  Analysis,  perform  other  appropriate  code  analyses 
that  might  have  potential  safety-critical  impacts,  such  as: 

-  Timing  analysis  -  for  safety-critical  hard  real  time  requirements, 
using  appropriate  static  or  dynamic  code  analysis  tools  to  analyze 
the  worst  case  execution  time  (WCET). 

-  Interrupt  analysis  -  analysis  of  the  coordination  of  interrupt 
handling  with  interruptible  and  non-interruptible  safety-critical 
processing. 

-  Algorithm  correctness  -  analysis  of  the  correctness  of  the 
implementation  of  any  safety-critical  algorithm(s).. 

-  Data  structure/usage  analysis  -  analysis  of  the  structure  and 
use  of  safety-critical  data  objects  associated  with  the  SSSF. 

-  OS  function  analysis  -  analysis  of  correct  use  of  OS  functions 
used  to  implement  LOR  1  functionality  for  the  SSSF. 
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Wrap  Up 
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Some  Key  Points 

•  Purpose  of  LoR  is  to  focus  and  manage 
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Some  Key  Points 

•  Purpose  of  LoR  is  to  focus  and  manage 

•  Software  FHA  should: 

o  Be  performed  as  early  as  reasonable 

o  “Rack  and  stack”  SSSFs  by  SwCI/LoR 

o  Identify  potential  redundancies  to  reduce  SwCI  1 
SSSFs 
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Some  Key  Points 


•  Purpose  of  LoR  is  to  focus  and  manage 

•  Software  FHA  should: 

o  Be  performed  as  early  as  reasonable 

o  “Rack  and  stack”  SSSFs  by  SwCI/LoR 

o  Identify  potential  redundancies  to  reduce  SwCI  1 
SSSFs 

•  Requirements  analysis  should  focus  on: 
o  Incompleteness 

o  Ambiguities 
o  Conflicts 
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Some  Key  Points 


Architecture  analysis  should  focus  on  weaknesses 
in  the  command  and  control  of  distributed  Safety- 
Significant  Software  Functions 


Some  Key  Points 


•  Architecture  analysis  should  focus  on  weaknesses 
in  the  command  and  control  of  distributed  Safety- 
Significant  Software  Functions 

•  Design  and  code  analysis  should  focus  on  Safety- 
Critical  Decision  Points  (can  the  internal  data  items 
used  by  the  software  be  corrupted  or  stale) 
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Some  Key  Points 


•  Architecture  analysis  should  focus  on  weaknesses 
in  the  command  and  control  of  distributed  Safety- 
Significant  Software  Functions 

•  Design  and  code  analysis  should  focus  on  Safety- 
Critical  Decision  Points  (can  the  internal  data  items 
used  by  the  software  be  corrupted  or  stale) 

•  In-Depth  Safety-Specific  Testing  should  be  derived 
from  the  analysis  results 
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Some  Key  Points 


•  Architecture  analysis  should  focus  on  weaknesses 
in  the  command  and  control  of  distributed  Safety- 
Significant  Software  Functions 

•  Design  and  code  analysis  should  focus  on  Safety- 
Critical  Decision  Points  (can  the  internal  data  items 
used  by  the  software  be  corrupted  or  stale) 

•  In-Depth  Safety-Specific  Testing  should  be  derived 
from  the  analysis  results 

•  All  analyses  and  testing  should  be  focused  on 
Causal  Factors  and  Mitigations 
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Best  Practices  for  the 
Architecture,  Design,  and 
Modernization  of  Defense  Modeis 

and  Simuiations 

Dr.  Katherine  L.  Morse,  JHU/APL 
Brian  Miiier,  US  Army  CERDEC  NVESD 
Michael  Heaphy,  OSD(AT&L)/DMSCO 
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NfSCO 


Madalini  aSImulBtlan  Caardlnatian  Offlct 


Outline 


•  Overview 

-  What  the  DMSRA  is  and  isn’t 

-  Goals/Vision/Motivation 

-  Composable  simulation  architecture 

•  Challenges 

-  Architectural  and  engineering 

-  Enterprise-wide  interoperability  and  reuse 

•  Best  practices  (patterns) 

-  Identified 

-  Planned  additions 

•  Conclusions 


lurcnn 


2 


Madalini  aSImulatlan  Caardinatian  Otfica 


Overview 


•  The  DMSRA  is  NOT  a  solution  architecture. 

•  It  establishes  a  vision  for  Defense  M&S: 

-  that  leverages  emerging  technologies,  and  enterprise  services; 

-  to  promote  reuse  and  interoperability. 

•  The  DMSRA  provides  broadly  applicable 
guidance. 

-  It  captures  principles,  standards,  and  best  practices  for  simulation 
architects  and  engineers  to  align  on  the  vision. 

-  It  is  not  mandatory. 
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DMSRA  Vision 


/  \ 

A  robust  modeling  and  simulation  (M&S)  capability  that  supports  a  full 

spectrum  of  DoD  activities  and  operations,  delivered  to  the  point  of  need, 
within  current  fiscal  constraints,  managing  schedules  and  risk  enabled  by 

agile  composition. 


•  Models  and  simulations  that: 

-  Are  modular  -  decomposed  into  loosely  coupled  reusable 
components; 

-  Execute  in  the  cloud  (where  practical)  -  hosted  in  the  cloud,  and  are 
capable  of  taking  advantage  of  cloud  characteristics  such  as  remote 
access  and  scalability: 

-  Adhere  to  enterprise-wide  composability  standards  -  follow 
standards  that  facilitate  the  reusability  of  components  across 
programs  and  Components. 
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OV-1  High  Level  Operational 
Concept  Graphic 


Vision 

Robust  modeling  and  simulation  (M&S)  supporting  a  full  spectrum  of  DoD  activities  and  operations,  delivered  to  the 
point  of  need,  within  current  fiscal  constraints,  managing  schedules  and  risk  enabled  by  agile  composition. 


Full  Spectrum  of  DoD 
Activities 

Delivered  to  the  Point 
of  Need 

Within  Current  Fiscal 
Constraints 

Agile  Composition 

I  Multi-  &  Inter-Domain  Support 

Leverage  Existing  M&S  Assets 

I  Consistency  of  Outputs  &  Outcomes 

Reduced  Lifecycle  Costs 

I  Leverage  DoD  IT  Infrastructure 

Reduced  Operational  Complexity 

Secure  Accessibility 


Realistic  Simulation  Environments 


Agility  &  Innovation 


Capabilities 


Remote  Execution 


Thin  Client  Access 


Modular  Components  1 1  SOA-based  Architecture 


Well-Documented  |^tandards-based  Interface^ 


Scalable 


Cross  Domain  Solutions 


Integration  With  Test/  Operational  Systems 


Accommodate  Occasional/ Sporadic  Connectivity 


lA/  Secure  Accessibility 


Authoritative  Data  Representations 


Enterprise  Tools/  Services 


DoD  IT  Infrastructure  and 
Architectures  (DoDIN) 


Policy 

Federal 

DoD 
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Composable  Enterprise 
Architecture  (EA) 


Architectural  and  Engineering 

Challenges 


•  Managing  a  hybrid  architecture  that  maintains 
interoperabiiity  with  iegacy  systems 

•  Decomposition  of  iegacy  systems  into  reusabie 
components 

•  Deveiopment  of  standards  to  faciiitate  composabiiity 
of  modeis 

-  Common  conceptual  model/framework  for  assembling 
components 

-  Verification  and  Validation  of  composed  simulations 
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Unique  M&S  Challenges  to 
Modular,  Open  System  Approach 


□  Account  Number 

□  Authentication 


□  Amount 

□  Currency 

□  Time  Stamp 


Get 
Balance 


The  bank  keeps  the  definitive  record 
of  the  amount  of  money  in  an 
account 


□  Observer  Coordinate, 

□  Target  Coordinate 


□  True/False 


The  terrain  database  is  a 
representation  of  the  terrain  based 
on  a  set  of  simplifying  assumptions; 

those  assumptions  affect  the 
suitability  and  accuracy  of  the  data 


Enterprise-wide  Interoperability 
and  Reuse  Challenges 


•  Implementing  governance  structures  that  enable  and 
encourage  modular,  open-systems  approaches 

*  Facilitating  trust  between  simulation  developers, 
dependent  upon  other  model  and  simulation 
developers  who  may  not  be  in  their  program  chain. 

-  This  will  require  simulation  program  managers  to  accept  some  risk 

-  It  will  also  require  adoption  of  common  conceptual  model  (s)  or 
frameworks 
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How  the  DMSRA  is  Addressing  the 

Chaiienges 

•  Collaborative  approach 

•  Leverage  existing  investments 

•  Develop  patterns  that  capture  best  practices, 
and  gaps  in  standards,  technology  and  practice 
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Collaborative  Approach 


M&S  COI  Architecture  Working  Group  (AWG) 

-  36  briefings  on  architecture  /  framework  initiatives 

-  Includes  briefings  from  all  4  Services,  MDA,  Joint  Staff,  and  NATO 

-  Domains 

♦♦♦  Training 


rnilSuite  FI  milBook 

B  mdWiki 

□  milWire 

CmHTube 

imCtASSinED//FOUO 

ES  mHSearch 

1 '  A  Account 

0  Support 

I 

FlmilBook 

Streams 

Content 

People 

Races 

@0  p.  ^ 

1 

Overview  Activity 

Content 

People 

Reports  Calendar  Content  Manager 

Actions About 

Manage- 

1 

❖  T&E 


>  Acquisition 
Experimentation 
C*  Analysis 


❖ 


Oniine  coliaboration 


Collaboration  and  coordinabon 
site  for  development  of  the 
Defense  Modeling  and 
SImulabon  Reference 
Architecture  (DMSRA) 

Owned  by: 

Kathenne  Morse,  Bnan  Miller, 
Michael  Heaphy 


'I  DMSRA  Introduction 

DMSRA  Patterns  (Enterprise  Capabilities,  Services) 
-n  DMSRA  Principles  (Rules  and  Best  Practices) 

Cl  DMSRA  Strategic  Purpose 
DMSRA  Technical  Positions  (Standards) 

1  OSD  References/Inpul 
>  USA  References/Input 
^  USAF  References/Input 
'  USMC  References/Input 
USN  References/Input 


@  Start  a  discussion 
g)  Write  a  document 
Q  Upload  a  file 
IS  Write  a  blog  post 
ill  Create  a  poll 
I ']  Create  a  status  update 

If  Invite  people  to  join  this 
group 

Q  Group  feeds 
CJ  Create  an  event 
f  Create  an  idea 


B  r 


Does  DMSRA  address  just  nin-ttme  or  the  full  lifecycle? 

SwMkaago  -  i-  ' 

Contributing  to  the  DMSRA 


-  Emphasizes  the  dynamic  and  coilaborative  nature  of  the  DMSRA 

-  Makes  the  revision  process  more  transparent 

-  Makes  it  easier  to  contribute  to  the  DMSRA 

-  Makes  contributions  immediateiy  avaiiable  and  easier  to  find 

-  https://www.miisuite.mil/book/groups/dmsra  (DoD  CAC  oniy) 

Madillnt  a  Slndlstlan  CMrdInilliR  onica 
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Leveraging  Existing  Investments 


•  The  DMSRA  effort  builds  on  the  Live,  Virtual, 
Constructive  Architecture  Roadmap  (LVCAR) 
principles: 

•  Do  no  harm 

•  Interoperability  is  not  free 

•  Start  with  small  steps 

•  Provide  central  management 

•  Other  investments  and  resources  leveraged: 

•  Defense  M&S  Glossary 

•  Verification,  Validation,  and  Accreditation  (VV&A)  Recommended 
Practices  Guide 

•  DoD  and  NATO  standards  references  and  tools 

•  Services’  architecture(s)  artifacts  and  practices 
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Patterns: 

Best  Practices  and  Gaps 


•  Extensibility  via  Patterns 

-  The  base  document  and  initial  patterns  were  not  sufficiently  comprehensive  to  meet 
the  DMSRA  vision 

-  Led  to  the  use  of  modular  patterns  to  extend  and  evolve  the  DMSRA  with  new 
technologies  and  associated  best  practices. 

•  DMSRA  Pattern  Outline: 

-  Pattern  overview:  Frames  topic  with  definitions,  technology  description, 
and  relevance  to  the  DMSRA 

-  Mapping  from  Capabilities,  and  Principles  and  Rules:  aligns  capability 
with  DMSRA  principles 

-  Pattern:  Provides  a  series  of  questions  the  user  should  ask  in  the  process  of 
deciding  whether  to  apply  the  technology/capability.  Documents  guidance  and  best 
practices  for  answering  the  questions  in  context  based  on  inputs  from  the  AWG. 

-  Technical  Positions:  Identifies  applicable  standards,  including  DoD 
adoption  status;  and  standardization  gaps 

-  References 
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Current  Patterns  Findings  (1  of  2) 


•  Cloud  migration 

-  Lower  overall  costs  to  the  consumer,  because  of  efficiencies 
obtained  by  pooling  much  of  the  computing  hardware  and  software; 

-  IT  functions  and  increased  flexibility  because  there  is  no  upfront 
investment  in  infrastructure  required  by  the  end  user 

•  Service-oriented  architecture 

-  The  Department  of  Defense  (DoD)  Chief  Information  Officer  (CIO) 
has  directed  the  DoD  to  leverage  commercial  SOA  technologies  to 
reduce  costs  and  increase  flexibility. 

-  This  pattern  aids  the  user  to  determine  the  suitability  of  an 
organizational  capability  for  migration  to  a  SOA  from  technical, 
programmatic,  and  domain  perspectives. 


lurcnn 
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Current  Patterns  Findings  (2  of  2) 


•  Decomposition  of  simulations  into  modular 
components 

-  Although  much  has  been  written  about  modular  simulation,  there  is  a 
gap  for  M&S-specific  standard  practices  for  decomposition. 

•  Verification  and  validation  of  modular 
components 

-  Cloud  computing  considerations:  The  hardware  and  operating 
system  the  simulation  is  hosted  on  are  out  of  the  control  of  the  user 
and  may  be  altered  from  the  configuration  used  during  validation 
without  the  user’s  knowledge. 

-  V&V  of  composed  simulations:  composition  of  validated  component 
models  does  not  ensure  a  valid  composed  simulation.  This  is  a 
known  gap  in  standards  and  practice. 


lurcnn 
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Way  Ahead 


Continue  collaborative  approach  to  capturing  best 
practices  in  patterns,  including  the  following  topics: 

*  Accommodating  occasional  /  sporadic  connectivity 

*  Cross  domain  solutions 

*  Distributed  simulation  and  federation  engineering 

*  Data 

*  Assessing  the  feasibility  of  remote  execution 

*  Gaming  architectures 

Continue  to  leverage  DoD  enterprise  architecture  and  IT 
capabilities  and  practices: 

*  Cloud  computing 

*  MOSA  and  SOA  practices  and  standards 
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QUESTIONS? 


POTOMAC  INSTITUTE 
FOR  POLICY  STUDIES 


Developing  Trust  for  a  Secure 
Microelectronics  Supply  Chain 


Dr.  Mike  Frit2e 


Senior  Fellow 


Potomac  Institute  for  Policy  Studies 


.V,  ■' 


Outline 


•  Articulation  of  Trust  Problem  (for  systems  folks) 


•  Measuring  Trust 


•  National  Strategy  for  Trust 
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Defense  Systems: 
Global  Supply  Chain 
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Increasing  complexity  in  the  supply  chain  results 
in  decreased  security  of  defense  systems 
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Microelectronics  Global  Supply  Chain 
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Threats  to  the  Hardware  Supply  Chain 


Hardware  threats  exist  throughout  the  global  microelectronics  supply  chain 


Research  & 
Design 

(Research,  Development, 
Prototyping) 

Un-vetted  3^^  party  IP 
increases  the  number  of 
people  with  knowledge  of 
a  design  and  provides 
opportunities  to  corrupt  a 
design 

Zero  Day  effects  can  be 
embedded  into  a  chip’s 
design,  go  undetected, 
and  be  triggered  after  a 
chip  has  been  produced 


The  Supply  Chain  —  From  design  and  production  to  deployment 
Malicious  insertions,  Counterfeits,  Clones,  Insider  Threat 


Production 

(Fabrication) 


The  U.S.  is  increasingly  relyin 
on  off-shore  foundries  to 
supply  components  for  our 
critical  mission  systems 
Only  2%  of  ASICs  used  in 
National  Security  Space 
systems  come  from 
DoD  trusted  foundries 
This  increases  the  risk  of 
malicious  insertion  to  include 
Trojan  horses.  Kill  Switches, 
and  Backdoors 


https:/ /www.bloomber 
g.com/ news/ articles/ 2 
008-10-01  /  dangerous- 
fakes 


Supply,  Stock 
and  Store 

(Testing  and  Verification, 
_ Acquisition) _ 


cqu 

.ck" 


Attack  vectors  exist 
throughout  the  entire 
supply  chain  to  include  - 
design,  fabrication, 
testing,  packaging, 
distribution,  and  end-of- 
life 

53%  of  counterfeit 
incidents  from  2003  — 
2013  were  for 
discontinued  (legacy) 
components 


Deployment 

(Deployed  mission 
systems.  Logistics  & 
Maintenance,  end-of-life) 


Insider  threats  and  counterfeits 

in  the  upgrade/ refresh  process 

Information  exploitation 

Electronic  warfare 

Kin  switches  and  backdoors 

can  be  used 

Poor  disposal  practices 
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Measuring  Hardware  “Trust” 

•  "‘Trust”  commonly  used  phrase  but  very  difficult  to  precisely 
and  quantitatively  define 

•  We  propose  an  ‘insurance”  based  definition  of  Trust 

T=R/M 

T  =  level  of  trust;  R  =  risk  mitigation  investment;  M  =  mission  value 


100%  trust  means  we  have  mission  “insured”  for  its  full  value 

Insurance  “purchased”  depends  on  value  of  mission  and 
nature  of  threats  of  interest 
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Relating  Risks  to  Threat  Type 


Hardware  Vulnerabilities 
created/ inserted  across  supply 
chain 


Existing  software  and 
hardware  vulnerabilities 
exploited 


Anyone  can  hack  software.  It  takes  a  nation  state  to 
attack  hardware 

Hardware  attacks  are  by  Nation  state  actors,  capable  of 
insertions  across  the  supply  chain;  requires  significant 
resources  and  expertise. 


Goals:  Strategic  political^  economic^  military  dominance 

Creates  vulnerabUities  usmg  Ml  spectrum 


Professional  or  state  sponsored  criminal  networks 
Goals:  Financial  gain.  Political  influence^  Information 

Discovers  unknown  vutnerabiUties 


Existential 


Existing 
software /human 
vulnerabilities 
exploited 


i 


w  Tiers 

f  l-ll 

r  -$ios- 

Goals:  Financial  gain 


Exploits  pre-exisVng  Known  vulnerabilities^ 


Nuisance 


DSB  Task  Force  Report:  Resilient  Military  Systems  and  the  Advanced  Cyber  Threat, 
http:/ / www.acq.osd.mil/ dsb/ reports/ 2010s/ResilientMilitarySystemsCyberThreat.pdf 
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Mitigation  Insurance: 
Impact  vs  Difficulty  Matrix 


Impact  of 
Attack 


Highest 

Risk 


Medium 

Risk 


LpOwcst 

Risk 


Attack  Difficulty 


Mitigation  “Insurance”  Goal: 

To  make  attacks  more  costly  (difficulty/ time /$)  for  the  attacker  than  the  defender 
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National  Strategy:  Address  the  entire  supply  chain 
US  Government  Solution  —  DMEA  Executive  Agent 


The  Trusted  Microelectronics  Supply  Chain 


Research  & 

_ K 

Supply, 

_ k 

- % 

Production 

> 

Stock  & 

> 

Deployment 

Design 

- K 

V 

Store 

V 

The  DARPA  solution  is  a  menu  of  hardware  security  options  that 
can  be  selectively  applied  to  tackle  known  security  threats 


JFAC  Organizational  Structure 


— 


DMribuInn  StcAemert  A  -  A«]pruved  fo^ 


Designate  DMEA  as  Executive  Agent 


Current  DoD  Policies  Include: 

•  Defense  Industrial  Base  Sector 
Specific  Plan  (2010) 

•  Mission  Assurance  Strategy 

(2012) 

•  Antiterrorism  Force  Protectior 

•  Counterfeit  Mitigation  Policies 
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National  Strategy: 

Rationalizing  &  Integrating  DoD  Capabilities 


The  Trusted  Microelectronics  Supply  Chain 


Research  & 
Design 


Production 


Supply, 
Stock  & 
Store 


Deployment 


Research  for  trusted  design  & 
manufacturing 
Manage  relationship  with 
Industry  for  R&D  access  to 
“pre-SOTA”  technology 
Focus  on  research,  design, 
and  prototyping 
Acquisition  of  IP  for  pre-fab 
research 


O 


fi 

i 

A 

SOTA 

1 

SOTP 

1 

T 

Legacy 

T 

Trusted 

i 

Trusted 

i 

Flexible 

Foundry 

Commercial 

Foundry 

(GF) 

Suppliers 
(79  Suppliers) 

(DMEA 

ASD(R&E) 

1 


SwA  Technical  HwA  Technical 
Working  Group  Working  Grouf 


Armed  Services  need  new 
policies  to  secure  deployed 
systems 
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Field  Programmable  Gate  Array  (FPGA) 

Assurance 


Raymond  C.  Shanahan 

Deputy  Director,  Anti-Tamper/Hardware  Assurance 
Office  of  the  Deputy  Assistant  Secretary  of  Defense  for  Systems 

Engineering 

20th  Annual  NDIA  Systems  Engineering  Conference 
Springfield,  VA  |  October  26, 2017 
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Electronics  as  a  Strategic  Issue 


Current  Tactical 


DoD  Trusted  Electronics  Issue 

*  Options  for  domestic  trusted  manufacture  of 
custom  DoD  electronics  are  diminishing 


FY03-present: 

DoD 

Trusted 

Foundry 

Program 


Larger 

Strategic 

Issue 


COTS  Electronics  Assurance  (DoD  &  Beyond^) 

•  Most  COTS  electronics  used  in  DoD  systems  are  pb  201 7: 

fabricated  overseas;  significant  risk  from  tamper  Trusted  & 

Assurod 

*  Risks  similar  for  the  broader  national  security  Micro- 
community,  banking,  critical  infrastructure,  etc.  electronics 


Access  to  Electronics  /  Electronics- 
based  economic  growth 

*  Shift  in  electronics  fabrication  creates 
potential  for  overseas  control 

*  End  of  Moore’s  Law  potential  carries 
economic  impacts 


Significant  electronics  challenges  represent  a 
strategic  level  national  issue 


PB  2018/ 

POM  19: 
Microelectronics 
Innovation  for 
National  Security 
and  Economic 
Competitiveness 


1  Including  the  broader  national  security  community,  banking,  critical  infrastructure,  commercial  industry,  etc. 
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Need  for 

Assured  FPGA  Functionality 


*  Commercial  FPGAs  are  in  widespread 
use  across  National  Security  Systems 
(NSSs)  in  embedded,  special  purpose 
applications 

-  Programmable  nature  of  FPGAs  and  System 
on  Chips  (SOCs)  make  them  vulnerable  to 
cyber  malware  and  malicious  insertion 

*  While  Application-Specific  Integrated 
Circuits  (ASICs)  have  performance  of  ten 
to  a  thousand  times  that  of  FPGAs, 
FPGAs  are  seen  as  achieving  custom 
hardware  performance  without  the  high 
manufacturing  cost  of  custom  ASICs 


FPGA  applications: 

-Communication 

systems 

-UAVs 

-Tactical  robotics 

-  Radar  systems 

-  Missile  control 

-  Satellites 
-Ships 

-Vehicle  control 
systems 

-Other 
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FPGA  Usage  by  Revenue  in 
Military/Aerospace  Sector 


Mil/Aero  Part  Types  by  Revenue  ($1,880M) 


ASICs,  $176, 9% __ 


MPUs,  $150,8% 


DRAM,  $300, 16%_^ 


Analog  ASSPs,  $399. 
,21% 


Data  Converters, 

$105,6% 

_  Power,  $69,4% 

.  Flash,  $42 , 2% 

SRAM,  $24 , 1% 

\  Logic  AS5P,  $23, 1% 
Other,  $58,  3% 


Source:  IDA  report,  Examination  of  DoD’s  Use  of  Microelectronics 
in  Weapon  Systems,  2013 
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Policy  Requirement  for 
Trust  vs.  Assurance 


*  There  is  no  policy  requirement  in  DoDI  5200.44  for  a  Trusted  FPGA  or 
other  COTS  product;  only  ASICs  as  follows: 

—  “In  applicable  systems,  integrated-circuit-related  products  and  services  shall  be 
procured  from  a  trusted  supplier  using  trusted  processes  accredited  by  the  Defense 
Microelectronics  Activity  (DMEA)  when  they  are  custom-designed,  custom- 
manufactured,  or  tailored  for  a  specific  DoD  military  end  use  (generally  referred  to  as 
application-specific  integrated  circuits  (ASIC)).” 

*  However,  there  are  policy  requirements  for  assurance  in  DoDI  5200.44,  to 
include  the  following  of  particular  relevance  to  FPGAs: 

—  “Mission  critical  functions  and  critical  components  within  applicable  systems  shall  be 
provided  with  assurance  consistent  with  criticality  of  the  system,  and  within  their  role 
within  the  system.” 

—  “Control  the  ...  security  of  software,  firmware,  hardware,  and  systems  throughout  their 
lifecycles,  including  components  or  subcomponents  from  secondary  sources.  Employ 
protections  that  manage  risk  in  the  supply  chain  for  components  or  subcomponent 
products  and  services  (e.g.,  integrated  circuits,  field-programmable  gate  arrays 
(FPGA),  printed  circuit  boards)  when  they  are  identifiable  (to  the  supplier)  as  having  a 
DoD  end-use.” 

—  “Detect  the  occurrence  of,  reduce  the  likelihood  of,  and  mitigate  the  consequences  of 
unknowingly  using  products  containing  counterfeit  components  or  malicious  functions 
in  accordance  with  DoDI  4140.67” 

—  “Detect  vulnerabilities  within  custom  and  commodity  hardware  and  software  through 
rigorous  test  and  evaluation  capabilities,  including  developmental,  acceptance,  and 
operational  testing” 
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Definitions  of  Trust  and 
Hardware  Assurance 


*  The  NDAA  FY2017  Sec.  231  trust  definition  beiow  reflects  DASD(SE)’s 
working  definition  of  the  term,  “hardware  assurance  (HwA)” 

—  The  other  trust  definition  below  is  used  by  the  DoD  Trusted  Foundry  Program 

*  Planned  update  of  DoD  Instruction  (DoDI)  5200.44  needs  to  add, 
clarify,  and  harmonize  the  definition(s)  of  HwA  and/or  trust 

—  Needed  to  eliminate  existing  confusion  in  the  community  between  what 
constitutes  trust  versus  HwA;  sometimes  referred  to  as  “big  T”  versus  “little  T” 
trust  or  assurance 

—  These  definitions  do  not  compete  with  one  another,  but  can  be  complementary  if 
integrated  and  harmonized  into  an  internally  consistent  definition  or  set  of 
definitions  within  DoDI  5200. 


2004  AT&L  Memorandum 

NDAA  FY2017  Sec.  231 

trust:  “the  confidence  in  one’s  ability  to 
secure  national  security  systems  by 
assessing  the  integrity  of  the  people  and 
processes  used  to  design,  generate, 
manufacture  and  distribute  national  security 
critical  components” 

trust:  “with  respect  to  microelectronics,  to  the 
ability  of  the  Department  of  Defense  to  have 
confidence  that  the  microelectronics  function  as 
intended  and  are  free  of  exploitable  vulnerabilities, 
either  intentionally  or  unintentionally  designed  or 
inserted  as  part  of  the  system  at  any  time  during 
its  lifecycle” 
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FPGA/SOC  Assurance  Risks 


Commercial  FPGA/SOC  security,  third 
party  intellectual  property  (3PIP),  and 
Electronic  Design  Automation  (EDA) 
tools  are  largely  unverified 

Industry  unlikely  to  invest  unless 
encouraged 

Some  Military/ Aerospace  and 
specialty  needs  are  not  being  met 

DoD  uses  FPGAs  heavily  in  critical 
systems  and  many  potential 
vulnerabilities  exist 

—  Potential  for  compromise  of  IP 

confidentiality  and/or  integrity,  or  EDA 
tool  integrity,  from  design  through 
deployment 

—  Inconsistencies  and  uncertainty/lack  of 
clarity  in  methods,  policy,  and 
enforcement 

—  Supply  chain  threat  and  vulnerability 
awareness  is  poor 


Errant 

Behavior 

2+2=10  ^ 


Malicious 
Modifications 


FPGA  System  On  Chip  (SOC) 


Insecure 
Components 


It' 

Side  Channels 


Counterfeits  /  Clones 
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Advancing  Hardware  Assurance 


Program  Protection  Plan 
Outline  &  Guidance 

•^TUIOKU- 


Systems  Engineering 


Policy 


•  DoD  Instruction 
(DoDI)  5000.02 

•  Program  Protection 
Plan  (PPP) 

•  International  Traffic 
in  Arms  Regulations 
(ITAR)  update  (in 
work) 


Joint  Federated 
Assurance  Center 


•  Software  assurance 
Know-how  &  tools 

•  Hardware 
assurance  Know 
how  &  tools 

•  Advanced  V  &  V 
capabilities 

•  Firmware 
Assurance  planning 


Trusted  &  Assured 
Microelectronics 


•  Access  to  state-of- 
the-art  foundries 

•  Trust  and 
assurance  methods 
and  demonstration 

•  Industrial  best 
practices  for 
assurance 

•  Implement  &  Demo 


COTS  and  FGPA 


•  Supply  chain  risk 
management 

•  FPGA  Assurance 
Strategy 

•  Radiation  hardened 
microelectronics 
initiative 
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Microelectronics  Trust 
Verification  Technologies 


Design  Verification 


•  Verification/assurance 
of  designs,  IP,  netlists, 
bit-streams,  firmware, 
etc. 


Physical  Verification 


•  Destructive  analysis  of 
ICs  and  Printed  Circuit 
Boards 


Functional  Verification 


•  Non-destructive 
screening  and 
verification  of  select  ICs 


DoD,  Intelligence  Community,  and  DoE  enhancing  capability 

to  meet  future  demand 
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FPGA/SOC  Lifecycle  Map 


FPGA/SOC  Hardware  and  Tool  Development 


FPGA/SOC  Provisioning  Operation  and 

Maintenance 


Packaging 
and  Test 


HWV&V  Vendor/End  User 


Hardware 

and 

Software 

Provisioning 


Physical  Construction  Analysis 
Non-Destructive  Physical 
Logical  Function  Physical 


Firmware  Design 


*  Assurance  risks  to 
FPGA/SOC  security, 
iP,  EDAtoois, 
practices,  and 
methods  necessitate 
mitigations  across 
the  iifecycle 


Design 

Verification 


Designer  IP 


Firmware  Verification 


Firmware  Development 


FPGA/SOC 

System 

Security 


System 
Integration 
and  test 


Sustainment 


Obsolescence 
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FPGA  Assurance  Strategy 

Overview 


•  DASD(SE)  continues  to  refine  the  strategy  to 
address  FPGA  assurance  risks  in  coordination 
with  the  Joint  Federated  Assurance  Center  (JFAC) 
HwA  Technicai  Working  Group  (TWG)  and  the 
Trusted  and  Assured  Microelectronics  (T&AM) 
program 

-  Leverage  existing  USG  and  industry  efforts  to  the  maximum 
extent  possible 

-  Promote  community  awareness  of  related  USG  efforts  via  a 
series  of  workshops  and  conference 

-  As  a  community,  continuing  to  identify  and  refine  the  portfolio  of 
assurance  efforts  to  focus  on  with  the  goal  of  synchronizing  and 
eliminating  stove-pipes  and  separate,  single-point  solutions  when 
possible 

-  Identify  gaps  and/or  activities  requiring  investment  and  elevate 
relevant  needs  to  the  JFAC  Steering  Committee  for  prioritization 
and  direction  regarding  resourcing 

-  In  particular,  align  with,  and  inform,  the  execution  plan  for  the 
T&AM  program 
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FPGA  Assurance  Strategy  has 
multiple  FPGA  Assurance  Focus  Areas 
across  the  FPGA  Lifecycle 


>- 
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FPGA/SoC 

FPGA/ 

FPGA/SoC 

Hardware 

Firmware 

Provisioning 

Development 

_  Development 

_ 

_ 

Supply  Chain  Threat 

Industry  Engagement 

Policy  and  Guidance 
(P&G)  and  Standards 


Independent  Verification 
And  Validation  (IV&V) 

New  HwA  Techniques 
and  Tools 


Operation  & 
Maintenance 


DoD  Specific  Needs 


Leverage  Related  Efforts 


Increase  availability  for  DoD  specific  needs  in  Military/Aerospace,  e.g.,  Strategic  Radiation- 
Hardened  (SRH)  technologies,  and  other  domestic  manufacturing  needs 


Coordinate  with  other  major  efforts  across  the  DoD,  Intelligence  Community  (1C),  the  broader 
United  States  Government  (USG),  industry,  and  academia.  For  example: 

•  Defense  Production  Act  (DPA)  Title  III  Trusted  FPGA 

•  Trust  in  FPGA  Studies 

•  Aerospace  Terms  of  Reference  (TOR)  related  to  assured  FPGA  and  ASIC  development 


Enhanced  interaction  with  the  1C  to  provide  more  specific  threat  information  to  enable  enhanced 
threat  assessment  and  vulnerability  analysis 


Engage  FPGA  manufacturers,  EDA,  3PIP,  and  other  vendors  to  facilitate: 

•  USG  IV&V  access  to  timely/detailed  supply  chain  information,  e,g.,  design,  chain  of  custody,  etc. 

•  Design  tool  and  3PIP  distribution  and  enterprise  usage 

•  Verification  features  in  the  design  or  that  are  enabled  by  the  design  tools 

•  Commercial  verified  and  validated  security  features,  EDA  tools,  3PIP  or  other  supply  chain  tools 


Develop,  contribute  to,  and/or  adopt  P&G  and  standards  that  promote  best  practices  across  DoD 
and  other  USG  acquisition  programs  as  well  as  industry  to  the  extent  possible 
•  Facilitate  use  of  commercially  viable/supportable  tools,  IP,  and  best  practices  where  possible 


Expand  JFAC  IV&V  capability  and  capacity  for  physical,  functional  and  design  V&V 

to  be  offered  to  clear  contractors  and  USG  acquisition  programs,  leverage  co-development,  data 

access,  design  for  assurance,  and  other  best  practices  to  enable  better  V&V  _ 


Develop  and  facilitate  the  transition  of  new  HwA  techniques  and  tools  to  verify  and  validate,  protect 
the  confidentiality  and  integrity  of,  and  gain  insight  into  the  chain  of  custody  of,  IP,  EDA  tools,  and 
the  FPGAs/SOCs  themselves 


J 

\ 
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FPGA  Activities  and  Investments 


ro 

1- 

01 

> 

O 


Inputs  V  Investments  &  Actions  ^  Outcomes 


Investment  Breakout 


•  Established  relationships  and 
programs  with  FPGA  and  other 
vendors 

•  JFAC  labs,  FFRDCs,  expertise 

•  Other  USG  partners  and 
commercial  suppliers 

•  Programs  and  COTS  parts 


EDA  tool  and  3PIP  V&V  development 

Physical  V&V  tool  and  capability 
development 

Security  and  assurance  architecture 
development 

Radiation  testing  and  validation 


•  USG  and  third  party  EDA  and 
3PIP  V&V  tools 

•  JFAC-assessed  FPGA  list,  IV&V 
tools,  vulnerability  assessments 

•  Counterfeit  and  SCRM  tools 

•  P&G,  standards,  and  best 
practices,  PPP 

•  SRH  assessments 


Cyber 
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IP  integrity  protection 
Access  enhancements 


Outreach  to  industry 


Enhanced  V&V  of 
State-Of-The-Art 
(SOT A)  FPGAs 


Commer- 
chI  ASIC 


Transition  activities 
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FPGA  Strategy  Outcomes 


Problem 

Actions  &  Investments 

Outcomes 

DoD  influence  is  limited  and 
national  security  needs  not 
satisfactory  for  required 
production  and  volume 


Potential  for  compromise  to 
confidentiality  and  integrity 
through  design  access, 
COTS  insertion,  and 
deployment  of  commercial 
FPGA  creates  risk  when 
USG  accesses  SOTA  FPGA 

DoD  uses  FPGAs  heavily  in 
critical  systems  and  many 
potential  vulnerabilities  exist 


Support  domestic, 
manufacturing  of  SOTA 
FPGAs  and  industrial 
engagement  for  USG  and 
strategic  growth  application 
areas,  including  radiation¬ 
hardening,  high  voltage,  etc. 

•  Evaluate  and  adopt  best 
practices,  and  specialized 
tools  and  services  to  assure 
integrity  and  confidentiality  of 
IP 


Provide  USG  HwA 
community  with  access  to, 
or  knowledge  of,  assured 
USG  IP,  3PIP,  EDA  tools, 
experts,  secure  computing, 
techniques,  etc.  for 
innovation 


•Availability  of  assured  SOTA 
FPGAs,  tools,  and  IP  for  USG 
acquisition  programs 


Enhanced  USG  access  to  assured 
SOTA  FPGAs,  IP,  and  EDA  tools 


Assurance  throughout  the  FPGA/ 
SOC  lifecycle  through  secure 
design  environments,  best 
practices,  V&V  and  supply  chain 
tools,  and  specialized  services 
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Leverage  Related  Efforts 


•  Trust  in  FPGA  Studies 

•  JFAC  HwA  TWG  efforts 

•  Defense  Production  Act  (DPA)  Titie  ili  Trusted  FPGA  Projects 

-  In  FY17,  DPA  Title  III  Phase  1  worked  with  FPGA  vendors  to 
develop  product  strategies  to  allow  USG  to  assure  FPGAs 

-  In  FY1 8,  Phase  2,  planned  start  of  implementation  of  those  product 
strategies 

•  Defense  Microeiectronics  Activity  (DMEA)  Trusted  FPGA 
Study 

-  Congressional  Add  to  engage  major  vendors 

•  Anti-Tamper  Executive  Agent-reiated  technoiogy 
deveiopment 

•  Printed  Circuit  Board  and  interconnect  Technoiogy  Executive 
Agent  technoiogy  deveiopment 
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Leverage  Related  Efforts  (cont’d) 


•  NSA/R2-sponsored  FPGA  Trust  &  Integrity  Research 

-  Aerospace  documenting  this  research.  Final  product  in  FY17 

•  Mission  Assurance  improvement  Workshop  (MAiW)  and 
Aerospace  Terms  of  Reference  (TOR) 

-  FPGA  assurance-related  TORs  for  design,  Trust  Assurance,  and 
SME  training  in  development 

-  Other  FPGA  and  ASIC  related  TORs  already  completed 

•  National  Defense  Industrial  Association  FPGA  Assurance  Workshops 

•  Intelligence  Advanced  Research  Projects  Activity  Trusted  Integrated 
Circuit  (TIC)  Phase  3 

-  FPGA  developed  using  split  fabrication 

•  Defense  Advanced  Research  Projects  Agency  programs 
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The  Way  Ahead 


•  Program  engagement 

-  Foster  early  planning  for  HwA  and  SwA,  design  with  security  and  assurance  in  mind 

-  Implement  expectations  in  plans  and  on  contract 

-  Support  vulnerability  analysis  and  mitigation  needs 

•  Community  collaboration 

-  Achieve  a  networked  capability  to  support  DoD  needs:  shared  practices, 
knowledgeable  experts,  and  facilities  to  address  malicious  supply  chain  risk 

•  Industry  engagement 

-  Communicate  strategy  to  tool  developers  and  develop  standards  for  common 
articulation  of  vulnerabilities  and  weaknesses,  capabilities  and  countermeasures 

-  Co-development  of  next  generation  COTS  with  DoD  capabilities  and  assurance 
considered 

•  Advocate  for  R&D 

-  HwA  and  SwA  tools  and  practices 

-  Strategy  for  trusted  microelectronics,  to  include  FPGAs/SOCs,  that  evolves  with  the 
commercial  sector 

•  People! 

-  Improve  awareness,  expertise  to  design  and  deliver  trusted  systems 


FPGA 
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Systems  Engineering: 
Criticai  to  Defense  Acquisition 


Defense  Innovation  Marketplace 

http:/ /WWW.  defenseinno  vationmarketplace.  mil 


DASD,  Systems  Engineering 

http://www.acq.osd.mil/se 
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For  Additional  Information 


Raymond  C.  Shanahan 

Deputy  Director,  Anti-Tamper/ 
Hardware  Assurance 

ODASD,  Systems  Engineering 

571  -372-6558 

raymond.c.shanahan.civ(g)mail.mil 


E-mail  -  osd.pentagon.ousd-atl.mbx.fpga-assurance@mail.mil 

JFAC  Portal  ~  https://jfac.army.mil 
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Engaging  the  DoD  Enterprise  to  Protect 
U.S.  Military  Technology  Advantage 


Brian  Hughes 

Office  of  the  Deputy  Assistant  Secretary  of  Defense 

for  Systems  Engineering 

20th  Annual  NDIA  Systems  Engineering  Conference 
Springfield,  VA  |  October  25, 2017 
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These  are  Not  Cooperative  R&D  Efforts 


Reaoer 

Yilong-I 


HUMVEE 


Dongfeng  EQ2050 


20^^  NDIA  SE  Conference 
Oct  25,  2017  I  Page-2 


Distribution  Statement  A  -  Approved  for  public  release  by  DOPSR.  SR  Case  #  18-S-0067  applies.  Distribution  is  unlimited. 


UNCLASSIFIED 


Case  History:  Titanium  Dioxide 


Waiter  Liew,  a  naturaiized  American  citizen,  business  owner,  and  technoiogy  consuitant 
stoie  DuPont’s  protocois  for  producing  its  superior  titanium  white  from  1997  through  2011 


•  DuPont  developed  $2.6B  per  annum  Titanium  Dioxide  business 
-  recognized  as  world  leader 

o  Processes  created  in  1940s  but  spent  $150M  year  to  improve  processes 
by  1  % 

■  Near  monopoly  on  the  manufacturing  techniques 
o  Shielded  its  titanium  dioxide  process 

■  Guards 

■  Escorted  Visitors 

■  Documents  and  blueprints  controlled 

o  Starting  in  1990’s  China  began  seeking  ways  to  illegally  acquire  DuPont’s 
methods 

■  China  accounts  for  approximately  25%  of  the  demand 


Liew  was  convicted  in  2014  on  each  of  twenty  counts  with  which  he  was  charged  and 
sentenced  to  serve  15  years  in  prison,  forfeit  $27.8  miiiion  in  iiiegai  profits,  and  pay 

$511,667.82  in  restitution 
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Bottom  Line  Up  Front 


•  Adversary  is  targeting  our  Controiled  Technical  Information  (CTI) 

•  DoD  is  emphasizing  protection  activities  to  encompass  the  full 
range  of  threats  and  vulnerabilities  across  the  acquisition  life 
cycle 

•  The  Joint  Acquisition  and  Protection  and  Exploitation  Cell 
(JAPEC)  enables  a  comprehensive  analysis  of  protections  for 
DoD’s  critical  programs  and  technologies  (CP&T)  and  addresses 
shortfalls 

•  Significant  amount  of  technical  expertise  resides  in  the  Defense 
Industrial  Base  (DIB) 

•  The  DIB  is  not  only  critical  to  protecting  that  information  but 
helping  DoD  identify  which  information  it  should  protect 


Partnership  between  DoD  and  DIB  is  vital 
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Agenda 

•  DoD  Efforts  to  Safeguard  Controlled  Technical 
Information  (CTI) 

•  Know  the  Environment 

*  Stakeholder  Dialogue 

*  Defense  Industrial  Base  (DIB)’s  Role  in  the  Process 
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Addressing  the  Loss  of  CTI 


Risk  =  f  ( threat,  vulnerabilities,  consequences) 

Goals: 

•  Enable  information-sharing,  coliaboration,  anaiysis,  and  risk 
management  between  acquisition,  Law  Enforcement  (LE), 
Counterinteiligence  (Ci),  and  inteiligence  Community  (iC) 

o  Connect  the  dots  in  the  risk  function  (map  blue  priorities,  overlay  red 
threat  activities,  warn  of  consequences) 

*  Integrate  existing  acquisition,  LE,  Ci,  and  iC  information  to 
connect  the  dots  in  the  risk  function  -  linking  blue  priorities  with 
adversary  targeting  and  activity 

o  Many  sources  and  methods  are  relevant  (e.g.,  HUMINT,  joint  ventures) 
o  Cyber  is  only  one  data  source 

•  Focus  precious  resources 

*  Speed  discovery  and  improve  reaction  time 

♦  Uitimateiy,  evolve  to  a  more  proactive  posture _ 
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JAPEC  Mission: 
Integrated  Analysis 


The  Joint  Acquisition  and 
Protection  and  Expioitation  Ceii 
(JAPEC)  integrates  and  coordinates 
anaiysis  to  enabie  Controlied 
Technoiogy  information  (CTI) 
protection  efforts  across  the  DoD 
enterprise  to  proactiveiy  mitigate 
future  losses,  and  exploit 
opportunities  to  deter,  deny,  and 
disrupt  adversaries  that  may 
threaten  US  military  advantage. 
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Identifying  Critical  Programs  and 
Technologies  for  Proactive  Protection 


ACQUISITION 

•  Identify  DoD’s  Critical  Acquisition 
and  Technology 

•  Link  technologies  across  the 
enterprise 

•  Identify  protection  methods 

•  Educate  the  workforce 


SECURITY 

•  I  nteg  rate  C  l/Secu  rity  postu  re 

•  Coordinated  Security  Classification 
Guides 

•  Onsite  protection  at  DIB 

•  Contractor  threat  education 


COUNTERINTELLIGENCE/ 

LAW  ENFORCEMENT 

•  Collect  against  adversary  activity 

•  Field  presence 

•  Facility  security  analysis 

•  Cl  threat  assessment 

•  Investigations  &  Prosecution 


Program 

Technology 

Capability 

Selection 


Define  Key  Areas  and 
Avenues  of  Risk 


EVALUATE 

-  Impact  of  Compromise 

-  Areas  of  Vulnerability 
,  -  Exploitation 


REQUIREMENTS 

Revise  requirements  based  on 
change  in  threat 


INTELLIGENCE 

•  Identify  adversary  technologies 
needs 


DIB 

•  Understand  Supply  Chain 

•  Proactive  approaches 

•  Improve  Information  Sharing  w/ 
DoD 


CIO/NETWORK  SECURITY 

Tiered  IT  security  controls 
Enroll  in  threat  sharing  forums 


JAPEC  projects  demonstrated  the  effectiveness  of  an  integrated  iterative  approach. 

JAPEC  methods  complement  other  DoD  efforts. 
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Agenda 


*  DoD  Efforts  to  Safeguard  Controlled  Technical 
Information  (CTI) 

*  Know  the  Environment 

*  Stakeholder  Dialogue 

*  Defense  Industrial  Base  (DIB)’s  Role  in  the  Process 
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Understanding  Your  Supply  Chain 


•  Increase  level  of  concern  for  DoD’s  protection  priorities 
throughout  the  suppiy  chain 

-  Includes  vendors,  mergers,  acquisitions,  subsidiaries 

•  Executive  Order  on  Assessing  and  Strengthening  the 
Manufacturing  and  Defense  Industriai  Base  and  Suppiy  Chain 
Resiliency  of  the  United  States  dtd  21  Juiy  2017 

•  Within  270  days 

-  (a)  identifies  military  and  civilian  materiel,  raw  materials,  and  other  goods 
essential  to  national  security; 

-  (b)  identifies  manufacturing  capabilities  essential  to  producing  goods  identified 
pursuant  to  subsection  (a)  of  this  section,  including  emerging  capabilities; 

-  (c)  identifies  defense,  intelligence,  homeland,  economic,  natural,  geopolitical,  or 
other  contingencies  that  may  disrupt,  strain,  compromise,  or  eliminate  supply 
chains  of  goods  identified  pursuant  to  subsection  (a)  of  this  section  (including 
as  a  result  of  the  elimination  of,  or  failure  to  develop  domestically,  capabilities 
identified  pursuant  to  subsection  (b)  of  this  section)  and  that  are  sufficiently 
likely  to  arise  so  as  to  require  reasonable  preparation  for  their  occurrence; 

-  (d)  assesses  resiliency  and  capacity  of  manufacturing  and  defense  industrial 
base  and  supply  chains  of  the  United  States  to  support  national  security  needs 


How  well  do  you  know  your  supply  chain? 
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Agenda 

•  DoD  Efforts  to  Safeguard  Controlled  Technical 
Information  (CTI) 

•  Know  the  Environment 

*  Stakeholder  Dialogue 

*  Defense  Industrial  Base  (DIB)’s  Role  in  the  Process 
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Dialogue  with  Protection 
Stakeholders 


•  Compliance  with  existing  rules  &  regulations  is  necessary  but 
not  sufficient 

-  Protection  is  more  than  completing  a  checklist 

•  What  is  crucial  to  your  organization  delivering  the  desired 
capability? 

-  Identify  who,  what  and  where  at  each  facility 

o  FSO  may  not  be  well  positioned  to  speak  to  this 

-  Are  there  links  with  other  programs,  especially  if  programs  are  in  a  different 
Military  Department? 

o  Informing  all  involved  parties  helps  focus  1C,  Cl,  and  LE  resources 

-  Are  there  plans  to  market  the  same  technology  to  other  Military 
Departments  or  Government  Agencies? 

o  Government  regulations  and  laws  protect  business  proprietary 

•  DoD/DIB  information  sharing  improves  the  US’  abiiity  to  focus 
priorities  on  most  critical  technoiogies 

-  Timely  reporting  to  DoD  which  includes  more  than  cyber  incidents 

-  Information  sharing  forums  enable  you  to  learn  from  other’s  experiences 


20^^  NDIA  SE  Conference 
Oct  25,  2017  I  Page-12 


Distribution  Statement  A  -  Approved  for  public  release  by  DOPSR.  SR  Case  #  18-S-0067  applies.  Distribution  is  unlimited. 


Agenda 

•  DoD  Efforts  to  Safeguard  Controlled  Technical 
Information  (CTI) 

•  Know  the  Environment 

*  Stakehoider  Dialogue 

*  Defense  Industrial  Base  (DIB)’s  Role  in  the  Process 
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DIB  Role 


•  Identify  crucial  elements  for  protection  up  front 

o  Requires  coupling  technical  know  how  with  CI/LE  expertise 

o  Develop  and  implement  training  that  focuses  specifically  on  CTI  handling  and  protection 
requirements 

•  Do  you  have  your  own  iist  of  technoiogies  cruciai  to  you? 

•  Report 

o  Cyber  incidents  o  Media  Theft  and  Loss 

o  Suspicious  contacts  o  Insider  Threats 

•  Consider  joining  the  DIB  CS  program 

o  Enables  Government  to  Industry  information  sharing 
o  Join  and  contribute  to  the  DIB  CS  program  at  http://dibnet.dod.mil/ 
o  Share  cyber  forensic  reports  with  DoD 

•  Maintain  an  open  diaiogue  with  aii  the  protection  stakehoiders 

o  Counterintelligence,  Law  Enforcement,  Network  Security,  etc. 

o  Targeting  U.S.  Technologies:  A  Trend  Analysis  of  Cleared  Industry  Reporting  at 
http://www.dss.mil/documents/ci/2017_CI_Trends_Report.pdf 


The  DIB  is  a  criticai  partner  in  preventing  unauthorized  access 
to  precious  U.S.  inteiiectuai  property  and  manufacturing  capabiiity  by  adversaries 
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Systems  Engineering: 
Criticai  to  Defense  Acquisition 


Defense  Innovation  Marketplace 

http:/ /WWW.  defenseinno  vationmarketplace.  mil 


DASD,  Systems  Engineering 

http://www.acq.osd.mil/se 
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Questions 


Mr.  Brian  D.  Hughes 

Director,  Joint  Acquisition  Protection  and 

Expioitation  Ceil  (JAPEC) 

brian.d.hughes3.civ@mail.mil 

571  -372-6451 
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Engineering 

Cyber  Resiiient  Weapon  Systems 

Melinda  K.  Reed 

Office  of  the  Deputy  Assistant  Secretary  of  Defense 
for  Systems  Engineering  (DASD(SE)) 

20th  Annual  NDIA  Systems  Engineering  Conference 
Springfield,  VA  |  October  25, 2017 
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Ensuring  Cyber  Resilience  in 
Defense  Acquisition  Systems 


Threat: 


-  Adversary  who  seeks  to  exploit  vulnerabilities  to: 

-  Acquire  program  and  system  information; 

-  Disrupt  or  degrade  system  performance; 

-  Obtain  or  alter  US  capability 

Vulnerabilities: 

-  Found  in  programs,  organizations,  personnel, 
networks,  systems,  and  supporting  systems 

-  Inherent  weaknesses  in  hardware  and  software  can 
be  used  for  malicious  purposes 

-  Weaknesses  in  processes  can  be  used  to 
intentionally  insert  malicious  hardware  and  software 

-  Unclassified  design  information  within  the  supply 
chain  can  be  aggregated 

-  US  capability  that  provides  a  technological 
advantage  can  be  lost  or  sold 

Consequences: 

-  Loss  of  technological  advantage 

-  System  impact  -  corruption  and  disruption 

-  Mission  impact  -  capability  is  countered  or  unable  to 
fight  through 


Access  points  are  throughout 
the  acquisition  lifecycle... 


MDD 


FRP 

A  A  AO 


Materiel 

Analysis 

Techndogy 
Maturation  & 

Risk 

Reduction 

Engineerings 
Manufacturing 
Develop  rite  nt 

Low- Initial 
Prffliuction 
(LRIP} 

Production  & 
Deployment 

Operations  S  Support 

OTSE 

Sustainment  Disposal 

...and  across  numerous  supply 
chain  entry  points 

-  Government 

-  Prime,  subcontractors 

-  Vendors,  commercial  parts 
manufacturers 

-  3’'^  party  test/certification 
activities 
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Key  Protection  Activities  to 
Improve  Cyber  Resiliency 


Program  Protection  &  Cybersecurity 

DoDI  5000.02,  Enclosures  3  &  14 


DoDM  5200.01,  Vol.  1-4  DoDM  5200.45  DoDI  8500.01' 

DoDI  5200.39  DoDI  5200.44  DoDI  5230.24  DoDI  8510.01 


Technology 

What:  A  capability  element  that 
contributes  to  the  warfighters' 
technical  advantage  (Critical 
Program  Information  (CPI)) 

Key  Protection  ActivitvU 

•  Anti-Tamper 

•  Defense  Exportability  Features 

•  CPI  Protection  List 

•  Acquisition  Security  Database 


Goal:  Prevent  the  compromise  and 
loss  of  CPI 


Components 

What:  Mission-critical  functions 
and  components 

Key  Protection  Activity: 

•  Software  Assurance 

•  Hardware  Assurance/Trusted 
Foundry 

•  Supply  Chain  Risk  Management 

•  Anti-counterfeits 

•  Joint  Federated  Assurance 
Center  (JFAC) 

Goal:  Protect  key  mission 
components  from  malicious 
activity 


Information 

What:  Information  about  the 
program,  system,  designs, 
processes,  capabilities  and  end- 
items 

Key  Protection  Activity: 

•  Classification 

•  Export  Controls 

•  Information  Security 

•  Joint  Acquisition  Protection  & 
Exploitation  Cell  (JAPEC) 

Goal:  Ensure  key  system  and 
program  data  is  protected  from 
adversary  collection 


Protecting  Warfighting  Capabiiity  Throughout  the  Lifecycie 


Policies,  guidance  and  white  papers  are  found  at  our  initiatives  site:  https://www.acq.osd.mii/se/initiatives/init_pp-sse.htmi 
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Program  Protection  and  Cybersecurity 
Reiationship  to  Key  Acquisition  Activities 


COCOMS 

IPLS 

S&T  IPLs 


Threats 


Intel 


JCIDS 


!•  TTRA 
ITA 

DIATAC 

STAR 

Others 


Operational  Needs 
Performance  Criteria 
Operational  Threats 


Program  Protection  Plan 
Outline  &  Guidance 


•  VERSION  J.O  • 
•July  2011  • 


PPP 


Deputy  Assistant  Secretaiy  of  Defense 

Systems  Engineering 


Jul  2011 


Strat/ 
Contract! 


TEMP 


-  Trusted  supplier  requirements 

-  Acquisition  reguiations  (Security,  Safeguarding 
Covered  Defense  Information,  Counterfeits,  etc.) 

-  Foreign/Internationai  Engagement 


-  Incorporation  into  technicai  baseiines 

-  SSE  entry  and  exit  criteria  in  SE  tech  reviews 

-  SSE  as  a  design  consideration 

-  Technicai  risks  and  mitigation  pians 


-  Data  needed  to  ascertain  cybersecurity 
requirements  are  met 

-  Cooperative  Vuinerabiiity  Assessments 

-  Adversariai  Assessments 


Security  Classification  Guide 
Counterintelligence  Support  Plan^ 
Criticality  Analysis 
Anti-Tamper  Plan  (If  Applicable) 
Cybersecurity  Strategy 


LCSP 


Informs  full  life  cycle  protection  activities  for 
the  program 

Lists  critical  components  that  require  attention 


Program  Protection  and  Cybersecurity  Considerations  Are 
integrated  in  AH  Aspects  of  Acquisition 
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Cybersecurity  Is  Everyone’s 
Responsibility 


DSni'atk  Forcr 


CYBER  SUPPLY 
CHAIN 


Navy 


Navy  CybT  Platform 


Army 


Significant  Efforts  by  Military 
Departments  to  Address  Cybersecurity 


Air  Force 


[  Cybtrreslllenct  effotts  need  to  extend  across  the 


UU’AKIMIM  OI  L>l»t\bt 

Dfcl  tNSE  SUtNCt  BOARD 


TASKFORCXR£]>ORT: 

Rebilient  Militar>'  Systems  and  iln 

r'mrKor  Th.-oa«  i 


© 


Cybersecurity  is  not  just  an  iT  / 
network  issue.  We  must  transiate 
Cyber  iT  /  Network  practices, 
standards,  etc.  into  physicai  system 
requirements. 


An  Opportunity  Exists  Across  the 
Services  to: 

•  Collaborate 

•  Mature  efforts,  and 

•  Move  toward  common  approaches 


Each  MILDER  is  moving  forward  to  meet  its  organizational  needs 
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Recommendations  from 
Defense  Science  Board 


Summary  of  Recommendations 


Five  categories  for  improvement 

1.  Understand  supply  chain  risk 

—  Expand  vulnerability  assessments 

2.  Mitigate  potential  vulnerabilities 

—  Improve  detection  and  reporting 

3.  Approach  acquisition  differently 
—  Enhance  program  protection  planning 
—  Improve  timeliness  of  supplier  vetting 
-  Improve  system  engineering 

—  Use  JFAC  and  JAPEC  effectively 

—  Consider  cybersecurity  Impact  of  COTS  products  and  components 

4.  Support  life-cycle  operations 

—  Establish  sustainment  PPPs  for  fielded  systems 
—  Collect  and  act  on  parts  vulnerabilities 

5.  Pursue  technical  solutions 


DSB  TASK  FORCE  ON  CYBER  SUPPLY  CHAIN 


Publicly-released  report  published  Feb  2017 
Available  at:  https://www.acq.osd.mil/dsb/reports/2010s/ 
DSBCyberSupplyChain ExecSummary Distribution A.PDF 
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Cybersecurity  in  Acquisition 


Department  of  Defense 
INSTRUCTIOX 

MLTiIBEil  5000.02 
January  7,  2015 

_ J’lfUiT; Qmgg  J,  e  Jtamari-  2JJ  7 

vmw 

SLrBJECT:  OperatHm  of  ItiE  Def^  A^-quiEilioai  Sj-iteia 
R£f£ii!ii£e£:  Se€Re&iEDces 


1.  “-'JEKiSE.  Tlij  Lnarudiifflii: 

1.  k  w-ffirHanK-fi  [Jie  !ni±iartt>'  id  DoD  Dirsciii'E  i'DqUDj/  50D0.J]<]  (Rfikixiice  (a))  anid 
DaDD  J2J40J  msaiei  flue  ktoim  DaD  kelractHMi  5000.02  i3!ikrfiDc.e  (jh)) 

tB  updale  edalJiihjeipdky  for  the  mnnagcmml  of  all  acqaisirkoi  aragrami  in.  accordflDffi  wilfa 
R£fen!n£e  (a),  tbe  giidpIlTtPS  of  OfiEce  of  MfluaEioDEau  and  Bndga  Ciicukf  A-1 1  (RefeHLCfi 
Cc)),  aoi  EpfrfBDfla  ttejgb 

I.  Aukcnkei  MikMwe  DedniEi  .^utbanAes  (^IDAs:)  to  tailoo:  t!ie  regdatiH^'  lequirEniEEiE 
and  ac-qijmtion  prncedinEE  in  ibi^  oiEtniotkiD  to  more  effideolLy  actdnie  pcoEtnin  olqeodi'eiv 
C'Cmu.tEni  nitb  blatkoiy  and  Rekienoe  (a). 

c.  rer/i^cez,  andjfrescrilwzprxaAirEjbrac^siiio/i  rEp!m:i?>rirCm  reicfedJlo 

Qi-ie-MrtTT^i-  ffl  me  JrfUfijnfln 

d.  JncifTpcrEtes  jjaf  canca'^  Unvcihv-i^jv  AJetHorai^m  I  7-Wj  /riyy. 


2.  APPOCAEmn".  Tki  instradioQ  to  OSD,  l!i£  ^filifaiy  DgaraoHiE.  (be  Offitje  of 

dia  ChaiimaE  of  Lhi  Joint  Cbdofi.  of  Staff  ijjt  Joiint  Staff  flifi  Combailani  CwninaDdE,  iIle 

CffizEoftbelnsjHctorCrennialofdijeDt^taiimmtofDgfeiue,  dm  Dtfmse  AgEn^ts.  diielioD 
Fiidd  Aitndtki^  and  all  o4bef  organizadonal  EnntifE  witbin  1ti£  DdD  (tefesed  to  i:olIooth<Ely  in 
dib  iiiBtnxijtiion  ai  tha  ~DoO  CooinDeins"^. 

3.  POOCY.  TTifl  oi’EtmcliinE  manaEtoDBit  pnnujjiias  and  mandatory  polkdBE  tbat  ®w€in  flifl 
Dafeue  AtquiEiiiffln  Syaem  are  -desaibed  in  EefaencE  (a]).  This  innroctioiipfOE’ides  dis 
dedaikd  procedumE  Ihat  guidn  die  opeadon  of  dm  systEtn. 


Acquisition  workforce  must  take  responsibility  for 
cybersecurity  from  the  earliest  research  and 
technology  development  through  system  concept, 
design,  development,  test  and  evaluation, 
production,  fielding,  sustainment,  and  disposal 

Scope  of  program  cybersecurity  includes: 

-  Program  information  Data  about  acquisition,  personnel,  planning, 
requirements,  design,  test  data,  and  support  data  for  the  system. 

-  Organizations  and  Personnel  Government  program  offices,  prime 
and  subcontractors,  along  with  manufacturing,  testing,  depot,  and 
training  organizations 

-  Networks  Government,  Government  support  activities,  and 
contractor  owned  and  operated  unclassified  and  classified 
networks 

-  Systems  and  Supporting  Systems  The  system  being  acquired, 
system  interfaces,  and  associated  training,  testing,  manufacturing, 
logistics,  maintenance,  and  other  support  systems 


Codified  in  DoDi  5000.02,  Enciosure  14,  Jan  26,  2017 
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Design  for  Cyber 
Threat  Environments 


Activities  to  mitigate  cybersecurity  risks  to  the  system 
include: 

•  Allocate  cybersecurity  and  related  system  security  requirements  to  the 
system  architecture  and  design  and  assess  for  vulnerabilities.  The  system 
architecture  and  design  will  address,  at  a  minimum,  how  the  system: 

1.  Manages  access  to,  and  use  of  the  system  and  system  resources. 

2.  Is  structured  to  protect  and  preserve  system  functions  or  resources,  (e.g.,  through 
segmentation,  separation,  isolation,  or  partitioning). 

3.  Is  configured  to  minimize  exposure  of  vulnerabilities  that  could  impact  the  mission, 
including  through  techniques  such  as  design  choice,  component  choice,  security 
technical  implementation  guides  and  patch  management  in  the  development 
environment  (including  integration  and  T&E),  in  production  and  throughout 
sustainment. 

4.  Monitors,  detects  and  responds  to  security  anomalies. 

5.  Maintains  priority  system  functions  under  adverse  conditions;  and 

6.  Interfaces  with  DoD  Information  Network  (DoDIN)  or  other  external  security  services. 


DoDI  5000.02,  Enclosure  14  establishes  a  threshold  for  what  to  address 
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Implementation:  Engineering 
Cyber  Resilient  Workshops 


Workshop  1  Findings 

1 .  Requirements  derivation  is  a 
challenge  area 

2.  Require  clarity  on  Risk 
Acceptance 

3.  Assessments  should  be 

Integrated  with  and  driven  by  SE 
Technical  Reviews  _ _ 


Workshop  2  Findinqs/Actions 

1 .  Definitions,  Taxonomy  &  Standards 
Framework 

2.  Knowledge  Repository 

3.  Consolidated  Risk  Guide 

4.  Assessment  Methods 


Workshop  3  Findinqs/Actions 

1 .  Establish  DAU  CRWS  CoP;  facilitate 
definitions,  taxonomy  standards 

2.  Develop  Risk,  Issues,  &  Opportunities 
engineering  cyber  appendix 

3.  Align  assessment  approaches 

4.  Explore  S&T  opportunities 

5.  Address  Workforce  needs _ 

6.  Industry 


5.  Needs  Forecasting 

6.  Industry  Outreach 


Workshop  4  (Aug  2017) 

Theme:  Changing  the  Culture  /  Method:  Leverage  existing  engineering  approaches 


•  Technical  Performance  Measures  and  Metrics 

-  Develop  Engineering  Guidebook 

-  Identify  TPMs  affected  by  Cyber  actions 

•  System  Engineering  Technical  Reviews 

-  Validate  that  existing  SETR  criteria  is  sufficient  for 
secure  and  resilient  system  design  and  sustainment 

•  Leveraging  System  Safety 

-  Identify  threshold  of  acceptable  risk 

-  Quantify  the  security-driven  risk 


*  Cyber  Resilient  Software 

-  Establish  an  outline  to  identify  engineering  design 
and  analysis  considerations  for  the  software  in  secure 
and  resilient  weapon  systems 

•  Risk,  Issues,  and  Opportunity  (RIO)  Guide 

-  Develop  appendix  for  Cyber  Risk 


Addressing  Recurring  Chaiienges: 

Design  GuideiineSj  impiementationj  Engineering  Assessment 
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NDIA  SE  Cyber  Resilient  Summit  and  Secure 

Weapon  System  Summit 
April  18-20,  2017 


NDIK 


WELCOME 

NDIA  Cyber  Resilient  &  Secure  Weapon  Systems  Summit 
April  18 -20th 


CYBER  RESILIENT  AND  SECURE 
WEAPON  SYSTEMS  ACQU'^^^^^^ 
PROPOSAL  DISCUSSION 


Integrated  C 


Covered  Defense  Information  -  In 


NDIA  Systems  Engineering  O 
&  Secure  Weapon  Syster 


Space  Resiliency  -  Cyber 

April  18, 2017 

- . 


ICOodm 
OaWCWWcr-- 
Wa  Ptwidiie.  C.t 


Platform  Cybersecurity 


Richard  Massey  Technical  Fellow  CISSP*  / 

Boeing  Military  Aircraft 

APPROVED  FOR  “€CEAS£  Rm|umI  for  R««mm  of  I-' -  174»32^  EC  I 


Initial  Industry  Outreach  Aligned 
with  CRWS  Series 

-  Industry  implementation  lessons  learned 

-  Emphasized  need  for  consistency  across 
communities 

-  Discussed  approaches  to  risk 
acceptance 

-  Offered  thoughts  on  implementing 
safeguards  on  manufacturing  floor 

-  Offered  areas  for  improvements  to 
methods,  standards,  processes,  and 
techniques  for  cyber  resilient  &  secure 
weapon  systems 

-  Thoughts  on  addressing  sustainment 
challenges 


20th  NDIA  SE  Conference 
Oct  25,  2017  I  Page-10 


Distribution  Statement  A  -  Approved  for  public  release  by  DOPSR,  SR  Case  #  18-S-0074  applies.  Distribution  is  unlimited. 


Joint  Federated  Assurance  Center: 
Software  and  Hardware  Assurance 

•  JFAC  is  a  federation  of  DoD  software  and  hardware  assurance 

(SwA/HwA)  capabilities  and  capacities  to: 

-  Provide  SW  and  HW  inspection,  detection,  analysis,  risk  assessment,  and 
remediation  tools  and  techniques  to  PM’s  to  mitigate  risk  of  malicious 
insertion 

•  JFAC  Coordination  Center  is  developing  SwA  tool  and  license 

procurement  strategy  to  provide: 

-  Enterprise  license  agreements  (ELAs)  and  ELA-like  license  packages  for  SwA 
tools  used  by  all  DoD  programs  and  organizations 

•  Initiative  includes  coordinating  with  NSA’s  Center  for  Assured  Software  to  address 
potential  concerns  about  the  security  and  integrity  of  the  open  source  products 

-  Automated  license  distribution  and  management  system  usable  by  every  engineer 
in  DoD  and  their  direct-support  contractors 

•  Lead  DoD  microelectronic  hardware  assurance  capability  providers 

-  Naval  Surface  Warfare  Center  Crane 

-  Army  Aviation  &  Missile  Research  Development  and  Engineering  Center 

-  Air  Force  Research  Lab 


Moving  Towards  Full  Operational  Capability 
JFAC  Portal:  https://jfac.army.mil/  (CAC~enabled) 
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US  Microelectronics 
Security  and  Innovation 


□  — 

-A'  ^ 

'u—* * 
Secure  loT 


Strategic  National  Security  Ap^cations 


Financial  &  Autonomous  Robust  +  Agile  Commercial  Space  Biomedical 
Data  Analytics  Systems  +  Al  Communicators 


Strategic  National  Economic  Competitiveness  Applications 

-d  !- 


proactive 
Awareness  & 
Security 

•  Supply  Chain  track 

•  Proactive 
Authorities 

•  Intelligence  &  Cl 

iM 


I 


.IM 


t 


Access  & 
Ass  ill  ranee 

'  Secure  Design 
'  IP,  EDA,  experts 
'  Found  y  assujjefl 
Access 
'  ProtetypQ 
Demonstrations 

TT 


I 


'E 


1  r 


Enabling 
Manufacturing 

•  SoP  Back-end 
parity  with  SotA 

•  SotA  on  200mm  I 
tools  at  SoP  I 

•  Mini  fabrication  for 
-ifiix  low  vol. 


n 


3  E 


5^ 


E 


I 


3  E 


3E 


Incentives  & 
arket  Growth! 

•  Acquisition  reform 
&  incentives 

•  Tax,  policy, 
regulation  reform 

•  R&D  and  ^cjmjGjstict 
fab  incentives 


Strategic 
Alliances 

•  Cooperative  ^D 

•  T rade  &  FMS 

•  Americas 

•  Europe 

•  Asia  partners 


5^ 


y 


i'ii  L- 


!i  Cl 


Disruptive  Research  &  Development 


1 


Materials,  devices,  circuits 


Architectures 


Design  tools  for  Complexity 


Experts,  Infrastructure,  Venture  Capital 


Science  &  Technology,  R&D 
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These  Are  Not  Cooperative 

R&D  Efforts 


U.S.  HUMVEE 


China’s 

Dongfeng  EQ2050 
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Protecting  DoD’s  Unclassified 

Information 


Contractor's  Internal  System 

System  Operated 

Security  requirements  from 

on  Behalf  of  the  DoD 

NIST  SP  800-171,  DFARS 
Clause  252.204-7012,  and/or 
FAR  Clause  52.204-21  apply 


Federal 

Contract 

Information 


Covered 

Defense  Information 
(includes  Unclassified 
Controlled  Technical 
Information) 


Controlled  Unclassified  Information 


Controlled  Unclassified 
Information 


Cloud  Service  Provider 


When  cloud  services  are 
used  to  process  data  on  the 
DoD's  behalf,  DFARS  Clause 
252.239-7010  and  DoD  Cloud 
Computing  SRG  apply 


Security  requirements 
from  CNSSI 1253,  based 
on  NIST  SP  800-53,  apply 


DoD  Owned  and/or 
Operated  Information  System 


Cloud  Service  Provider 


When  cloud  services  are 
provided  by  DoD,  the  DoD 
Cloud  Computing  SRG  applies 
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Contract  Regulation  for  Safeguarding  Covered 

Defense  Information 


Implementaitioti  of  NIST  SP  800*171  —  What  Happetis  on 
December  31,2017? 


In  response  to  the  December  31, 2017  implementation  deadline,  companies 
should  have  a  system  security  plan  In  place,  and  associated  plans  of  action 
to  address  any  security  requirennents  not  yet  implemented 

If  Revision  1  of  NIST  SP  eoO-171  was  not  "In  effect"  when  the  contract  was 
solicited  j  the  contractor  should  work  with  the  contracting  officer  to  modify 
the  contract  to  Include  NIST  SP  aOO-171,  Revision  1  (Dec  2016] 

-  DoD  guida  nee  is  for  contracting  officers  to  work  with  contractors  who 
request  assistancein  working  towards  consistent  implementation  of  the 
latest  version  of  DFAR5  Clause  252,204^7012  and  NI5T5P  SOO-171 
The  contractor  self-attests  (by  signing  contract)  to  becompilant  with  OFARS 
Clause  252,204-7012,  to  inciude  Implementation  of  NISTSP  SOO-171  (which 
allows  for  planned  implementation  of  some  requirements  if  documented  in 
the  system  security  plan  and  associated  plans  of  action) 

The  solicitation/contract  may  allow  the  system  security  plan,  and  any 
associated  plans  of  action,  to  be  incorporated,  by  reference,  into  the 
contract  (e,e„  via  Section  H  special  contract  requirement) 


Cybersecurity  in  DoD  Acquisition  Regulations  page: 


on 


Cybersecurity  Challenges 


^  Protecting  DoD’s  Unclassified  Information 


DFARS  Clause  252.204-7012, 
Safeguarding  Covered  Defense 
Information  and  Cyber  Incident 
Reporting,  published  Oct  2016 


Industry  information  Day,  June  23, 2017 


Purpose: 

■  Establish  minimum  requirements  for  contractors  and 
subcontractors  to  safeguard  DoD  unclassified  covered  defense 
information  and  report  cyber  incidents  on  their  contractor  owned 
and  operated  information  systems 

Contractor  is  required  to: 

■  Implement  NIST  SP  800-171  Controls  for  unclassified  non-Federal 
Information  Systems 

■  Report  cyber  incidents  affecting  covered  defense  information 

■  Submit  malware  when  discovered 

■  Submit  media  when  requested  by  DoD 

■  Flow  down  Clause  to  subcontractors  when  covered  defense  information  is 
subcontractor  networks 


http://dodprocurementtoolbox.com/  for  Related  Regulations,  Policy,  Frequently  Asked  Questions,  and  Resources 
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Cybersecurity  for  Advanced 
Manufacturing  Systems 


Operational  Technology  NDIK 

Environment 

ICS  systems  are 
long-lived  capital 
investments 
(15-20  year  life) 

“Production 
mindset"  with  little 
tolerance  for  OT 
down  time 


Technical  data  flowing  through  the  system  is  highly  valued  by  adversaries 


NDIA  Cybersecurity  for  Advanced  Manufacturing  Joint  Working  Group  April  20,  2017 


Challenges  In  DoD  and  the  Manufacturing  Environment  are  Cross  Cutting 


Nascent 
cybersecurity 
awareness  and 
limited  workforce 
training 

Manufacturing 
jobs  bring 
executable  code 
into  system 


20th  NDIA  SE  Conference 
Oct  25,  2017  I  Page-16 


Distribution  Statement  A  -  Approved  for  public  release  by  DOPSR  on  MM/DD/201 6,  SR  Case  #  1 6-S-1 757  applies.  Distribution  is  unlimited. 


Cyber  Community  of  Interest 
Roadmap  Key  Capability  Areas 
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Assuring 

Effective  Missions 


Assess  and  control  the  cyber  situation  in  mission  context 


Agile 

Operations 


Dynamically  reshape  cyber  systems  as  conditions/goals  change,  to 
escape  harm 


Resilient 

Infrastructure 


Withstand  cyber  attacks,  and  sustain  or  recover  critical  functions 


Trust 


Establish  known  degree  of  assurance  that  devices,  networks,  and 
cyber-dependent  functions  perform  as  expected,  despite  attack  or 
error 
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(MSE  &  EMT)  cross-cutting  areas  in  analysis  of  Joint  Chiefs  of  Staff  Cyber  Gaps 
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Program  Protection  and  Cybersecurity 
in  Acquisition  Workforce  Training 


•  ACQ  160:  Program  Protection  Overview 

-  Distance  learning  (online);  ~3  days 

-  Provides  an  overview  of  program  protection  concepts,  policy  and  processes, 
includes  overview  of  DFARS  252.204-7012 

-  Intended  for  the  entire  Acquisition  Workforce,  with  focus  on  ENG  and  PM 

-  Course  deployed  on  DAU  website  on  15  Aug  2016 


ENG  260:  Program  Protection  Practitioner  Course  (est.  depioyment 
Summer  2018) 

-  Hybrid  (online  and  in-class);  ~1  week 

-  Intended  for  Systems  Engineers  and  System  Security  Engineers 

-  Focuses  on  application  of  program  protection  concepts  and  processes,  including 
PM  responsibilities  for  implementing  DFARS  252.204-7012 


I&su 


Effective  program  protection  planning  requires  qualified,  trained  personnel 
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Summary 


•  Each  system  is  different;  approaches  must  be  taiiored  to  meet  the 
requirement,  operationai  environment  and  the  acquisition 

-  We  will  embed  cybersecurity  risk  mitigation  activities  into  the  acquisition 
program  lifecycle 

•  We  must  bring  to  bear  poiicy,  toois,  and  expertise  to  enabie  cyber  resiiiency 
in  our  systems 

-  Translate  IT  and  network  resiliency  to  weapon  system  resiliency 

-  Establish  system  security  as  a  fundamental  discipline  of  systems  engineering 

•  Opportunities  for  government,  industry  and  academia  to  engage: 

-  How  can  we  thoughtfully  integrate  cybersecurity  practices  in  existing 
standards  for  embedded  software? 

-  How  can  we  better  integrate  program  protection  and  cybersecurity  risks  into 
program  technical  risks? 

-  Can  we  establish  system  requirements  that  restricts  a  system  to  a  set  of 
allowable,  and  recoverable  behaviors? 

-  How  can  we  carefully  engineer  stronger  resiliency  in  systems  that  are  being 
modernized? 
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Systems  Engineering: 
Criticai  to  Defense  Acquisition 


PP/SSE  Initiatives  Webpage 

http://www.acq.osd.mil/se/initiatives/init_pp-sse.html 


JFAC  Portal 

https://jfac.army.mil/  (CAC-enabled) 
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For  Additional  Information 


Ms.  Melinda  Reed 
ODASD,  Systems  Engineering 

571  -372-6562 

melinda.k.reed4.civ(g)mail.mil 
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Program  Protection  and 
Cybersecurity  in  DoD  Policy 


DoDI  5000.02  Operation  of  the  Defense  Acquisition  System 

-  Assigns  and  prescribes  responsibilities  for  Cybersecurity,  includes  security,  to  the  acquisition 
community 

-  Regulatory  Requirement  for  Program  Protection  Plan  at  Milestones  A,  B,  C  and  FRP/FDD;  PM  will 
submit  PPP  for  Milestone  Decision  Authority  approval  at  each  Milestone  review 


DoDI  5200.39  Critical  Program  Information  Identification  and  Protection  Within 

Research,  Development,  Test,  and  Evaluation 

-  Establishes  policy  and  responsibilities  for  identification  and  protection  of  critical  program  information 

-  Protections  will,  at  a  minimum,  include  anti-tamper,  exportability  features,  security,  cybersecurity,  or 
equivalent  countermeasures. 

DoDI  5200.44  Protection  of  Mission  Critical  Functions  to  Achieve  Trusted  Systems  and 

Networks 

-  Establishes  policy  and  responsibilities  to  minimize  the  risk  that  warfighting  capability  will  be  impaired 
due  to  vulnerabilities  in  system  design  or  subversion  of  mission  critical  functions  or  components 

DoDI  4140.67  DoD  Counterfeit  Prevention  Policy 

-  Establishes  policy  and  assigns  responsibility  to  prevent  the  introduction  of  counterfeit  material  at  any 
level  of  the  DoD  supply  chain 

DoDI  8500.01  Cybersecurity 

-  Establishes  the  DoD  Cybersecurity  Program,  the  DoD  Principal  Authorizing  Official  and  Senior 
Information  Security  Officer  to  achieve  cybersecurity  through  a  defense-in-depth  approach  that 
integrates  personnel,  operations,  and  technology 
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Defense  Research  &  Engineering 

Strategy 


Mitigate  current  and  anticipated  threat  capabilities 


Enable  new  or  extended  capabilities  affordably 
in  existing  military  systems _ 

Create  technology  surprise  through  science 
and  engineering 


Focus  on  Technical  Excellence 
Deliver  Technologically  Superior  Capabilities 
Grow  and  Sustain  our  S&T  and  Engineering  Capability 
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Evolving  Capability 


•  Up  until  World  War  II,  almost  all  munitions 
missed  the  mark 


-  Massing  of  forces  needed  to  achieve  effects 

•  Strategic  government  investments  created  an 
“offset”  providing  technological  advantage 

-  Atomic  weapons,  precision  guided  munitions  aiiow 
reiiabie  targeting 

-  Massing  of  forces  no  ionger  absoiute  necessity 


•  Current  innovations  are  driven  by  industry 

-  Broadiy  avaiiabie  technoiogy  creates 
a  need  for  veiocity 
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Systems  Are  Changing 


From: 

To: 

•  Systems  built  to  last 

•  Systems  built  to  evolve 

•  Heuristic-based  decisions 

•  Data-driven  decisions 

•  Deepiy  integrated  architectures 

•  Layered,  modular  architectures 

•  Hierarchicai  deveiopment 
organizations 

•  Ecosystems  of  partners,  agile 
teams  of  teams 

•  Satisfying  requirements 

•  Constant  experimentation  and 
innovation 

•  Automated  systems 

•  Learning  systems 

•  Static  certification 

•  Dynamic,  continuous  certification 

•  Standalone  systems 

•  Composable  sets  of  mission 
focused  systems 

Systems  Engineering  Needs  to  Change 

Credit:  Derived  from  David  Long,  Former  INCOSE  President 
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Industrial  Age  Acquisition  and 
Engineering  Processes 


Taylor’s  scientific  management 

-  Empirical  methods  to  synthesize  workflows 
to  Improve  economic  efficiency 

-  Inspires  industrial  and  systems  engineering, 
business  process  management,  lean  six  sigma, 
operations  research 


Optimizing  engineering  &  production 
drives  need  for  stable  requirements, 
weli-defined  processes 

Optimizing  methods  to  change 
engineering  &  production  requires 
increasing  the  cycles  of  learning: 

-  To  identify  necessary  changes 

-  To  incorporate  those  changes  into  systems 


Materiai  Soiution  Anaiysis 


P 


Technicai  Maturity  and  Risk  Reduction 


P 


Engineering  and  Manufacturing  Deveiopment 


P 


Limited  Rate  Production 


P 


Operationai  Testing  => 

Fuii  Rate  Production  =>  Fieiding 
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Initiatives  to  Accelerate  Change 


•  National  Defense  Authorization  Act  (NDAA)  for  Fiscal 

Year  2017  Acquisition  Agility  Act 

-  Modular  Open  Systems  Approaches 

-  New  authorities  for  prototyping,  experimentation  &  rapid  fielding 

-  Defining  requirements  likely  to  evolve  due  to  evolving  technology, 
threat  or  interoperability  needs 

•  Reorganization  of  USD(AT&L)  -  NDAA  FY2017 

-  Creates  separate  organizations  for  acquisition  and  for  innovative 
technologies 

•  Middle  Tier  Acquisition  Policy  -  NDAA  FY2016 

-  Creates  alternate  acquisition  path  for  rapid  prototyping  and  fielding 

•  Engineered  Resilient  Systems  -  201 1 

-  Research  and  development  of  deep  tradespace  analysis  methods 
to  address  the  nature  of  evolving  missions  and  threats 

•  Joint  Urgent  Operational  Needs  processes  -  2004 
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Methods  for  Managing 
Software-Intensive  Acquisitions 


Spiral  Development 
Model  (Boehm  1986) 


Incremental  Commitment 
Model  (Boehm  2007) 


DoD  Instruction  5000.02  -  Operation  of  the 
Defense  Acquisition  System  (Jan  2015) 


\  / 

0 


A _ ^ 
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Agile  Development  -  2001 
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other  Systems  Engineering 

Perspectives 


•  MIL-STD-499  Engineering  Management 

-  Issued  by  Air  Force  in  1 969  and  1 974 

■  Draft  MIL-STD-499B  never  published  in  1990’s  acquisition 
reform  era 

-  Not  time-sequenced,  like  the  V-model 

-  Process  seems  to  encourage  trades  in  the 
“need-space”  and  the  “solution-space” 

-  Less  focused  on  production  ^ - 


Less  prescriptive  useful 
in  organizing  activities 


PROCESS  OUTPUT 


Figure  1-3.  The  Systems  Engineering  Process 
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Methods  for  Selecting 
Acquisition  Approaches 


Notes: 

•  Framework  helps 
overcome  tendency  to 
develop  optimal 
solutions  to  static 
requirements 

•  Each  axis  belongs  to  a 
separate  community 

•  Uncertainty  around 
Requirements  and 
Technology  can  be 
informed  by  intelligence 
community 


Credit:  Derived  from  Michael  Pennock,  Stevens  Institute 
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Interesting  Research  Questions 


*  Gauging  confidence  in  requirements,  abiiity  to  respond 

*  Anaiysis  of  trades  across  the  mission  space  and  the 
soiution  space 

*  Gauging  risk,  rework 

*  Hedging  methods 

*  Actuai  increases  in  velocity  of  capability  delivered 

*  Methods  to  increase  ability  to  respond 

-  e.g.,  MBSE,  advanced  manufacturing 

*  Dynamic  and  continuous  learning  and  certification 

*  Multiple  systems  interrelationships 

-  Portfolio  management,  mission  engineering 

*  Others? 
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For  Additional  Information 


D.  Scott  Lucero 

Deputy  Director,  Strategic  Initiatives 

Office  of  the  DASD 
Systems  Engineering 

571-372-6452  |  don.s.lucero.civ@mail.mil 
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Systems  Engineering: 
Criticai  to  Defense  Acquisition 


Defense  Innovation  Marketplace 

http:/ /WWW.  defenseinno  vationmarketplace.  mil 


DASD,  Systems  Engineering 

http://www.acq.osd.mil/se 
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Modeling  the  Digital  System  Model 
(DSM)  Data  Taxonomy 

Philomena  Zimmerman 

Office  of  the  Deputy  Assistant  Secretary  of  Defense 

for  Systems  Engineering 

20th  Annuai  NDiA  Systems  Engineering  Conference 
Springfieid,  VA  |  October  25,  2017 
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Agenda 


•  DSM  Data  Taxonomy  Overview 

•  Evolution  of  the  DSM  Data  Taxonomy 
(Tabular,  Mind  Map,  SysML) 

•  Modeling  the  DSM  Data  Taxonomy 

•  Benefits 

•  Path  Forward 
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DSM  Data  Taxonomy  Overview 


Purpose 

Provides  a  model  to  aid 
programs  in  defining  an 
authoritative  source  of  truth 

Builds  an  integrated  taxonomy 
providing  stakeholders  an 
organized  structure  for  the  types 
of  technical  data  to  be 
considered  across  the  life  cycle 

Establishes  a  Common 
Vocabulary  that  can  be  used  by 
all  programs 


A  change  from  document  centric  taxonomy  in  DAG  CH  3-4. 1.7 
Technical  Data  Management  Process. 


Use  as  a  basis  to  drive  the  community  towards  Digital  Engineering 
across  disciplines,  systems  and  enterprises  to  support  life  cycle  activities  from 

concept  to  disposal. 
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DSM  Intended  Use 


Defines  the  broad  categories 
of  data 


Provides  the  program ’s  DE 
ecosystem 

DE  Ecosystem 

Requirements 

Contracts  j  I  1  ^  Architecture 


Cost 


Management 


Design 


Manufacturing 


Configuration 

Management 


Component  of  the  DE 
Infrastructure 


Identifies  the  data  and  data 
rights 


DSM  Data  Taxonomy  provides  the  broad  categories  of  data  that  shouid  be 

considered  across  the  iifecycie 
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Data  Taxonomy  Uses 


•  The  taxonomy  serves  as  a  common  vocabulary  for  enterprise 
and  program  consideration. 

•  Use  it  to  define  the  data  the  program  will  need  to  create  and 
manage. 

•  Use  it  to  determine  what  toois  wiii  use  or  produce  the  data. 

•  Use  it  to  determine  who  owns  and  controls  the  data  at  any 
point  in  time  in  a  programs  life. 

•  Use  it  to  identify  what  data  wili  be  deiivered  on  contract,  what 
format  the  data  should  be  received  in. 

•  Use  it  to  identify  what  data  has  associated  data  restrictions. 

•  Use  it  to  identify  what  data  needs  to  be  protected  and  handled. 

•  Use  it  to  define  the  data  that  beiongs  in  views,  digital  and  or 
other  artifacts. 
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Evolution  to  Modeling  the  DSM 

Data  Taxonomy 


r 


Mind  Mapping 
Tool 


A 


SysML 

Modeling  Tool 


•  Initial  attempt  to 
organize  and  construct  a 
hierarchical  structure  for 
technical  data  in  a 
system  from  documents 
and  guidelines  (e.g., 

DAG,  ICD,  CDD,  SEP, 
TEMP,  MIL-STD,  SME, 

V  etc.)  _  y 


•  Prototype  testing  using  a 
mind  mapping  tool  to 
visualize  hierarchical 
relationships  between 
system  components  and 
their  respective  digital 
artifacts 

V _ 


•  Utilized  a  System 
Modeling  Language 
(SysML)  modeling  tool 
to  construct  a 
hierarchical  structure 
and  enable  the  capture 
of  digital  technical  data 
for  use  and  reuse  in  a 
y  model _ 
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DSM  Data  Taxonomy  in  Excel 


1 

Definition  Info 

1 

1 

1.1 

Requirements  Info 

ISO 

A  requirement  is  a  statement  that  identifies  a  product  or  processes  operational,  functional,  or  design 

1.1.1 

Customer/Stakeholder  Needs/User  Info 

DI-IPSC-81431A/SEBOK 

Set  of  stakeholder  requirements  are  clarified  and  translated  from  statements  of  need  into  engineerir 

l.l.l.l 

Capability 

ICD 

A  capability  is  the  ability  to  achieve  a  desired  effect  under  specified  standards  and  conditions  throi 

l.l.l.l.l 

Capability  Gap 

ICD 

The  inability  to  execute  a  specified  course  of  action.  The  gap  may  be  the  result  of  no  existing  capabi 

1.1.1.1.2 

Required  Capabilities 

ICD 

A  capability  required  to  meet  an  organization's  roles,  functions,  and  missions  in  current  or  future  of 

1.1.1.1.3 

Enabling  Capabilities 

DoDD  3700.01 

services,  systems,  processes,  and  related  infrastructure  that  enable  the  exercise  of  authority  and  dii 

1.1.1.1.4 

Applicable  Joint  Capability  Areas  (JCAs) 

ICD 

JCAs  are  collections  of  similar  capabilities  logically  grouped  to  support  strategic  investment  decisic 

1.1.1.2 

Contract 

DI-IPSC-81431A/SEBOK 

1.1.1.3 

Operational 

DI-IPSC-81431A/SEBOK 

1.1.1.3.1 

Mission  Information 

JCIDS  products  (FAA,  FNA,  FSA);  ICD,  CDD,  OMS/MP) 

1.1.1.3.1.1 

Mission  Essential  Tasks 

Requirements  documents  (Operational  and  Functional  Concepts;  JCIDS  pt 

A  collective  task  a  unit  must  be  able  to  perform  successfully  in  order  to  accomplish  its  doctrinal  or 

1.1.1.3.1.2 

Mission  Objectives/Operational  Outcomes/Effects/Military  Objective  Achieved  Infc 

Requirements  documents  (Operational  and  Functional  Concepts;  JCIDS  pt 

Effectiveness  The  overall  degree  of  mission  accomplishment  by  a  system  under  realistic  conditions  ( 

1.1.1.3.1.2.1 

Concept  of  Operations  Summary 

ICD 

A  verbal  or  graphic  statement,  in  broad  outline,  of  a  commander's  assumptions  or  intent  in 

1.1.1.3.1.2.2 

Operational  Outcome 

ICD 

(d)  Identify  what  measurable  operational  outcomes  are  required;  what  effects  must  be  produced  to  < 

1.1.1.3.1.3 

Measures  of  Effectiveness  (MoE) 

CDD 

Measures  designed  to  correspond  to  accomplishment  of  mission  objectives  and  achievement 

1.1.1.3.1.4 

Measures  of  Suitability  (MoS) 

CDD 

Measure  of  an  item's  ability  to  be  supported  in  its  intended  operational  environment.  MOS's  typical 

1.1.1.3.2 

Threat  and  Operational  Environment  Info 

System  Threat  Assessment  Report  (STAR) 

1.1.1.3.2.1 

Operational  Environment 

System  Threat  Assessment  Report  (STAR) 

This  is  a  composite  of  conditions,  circumstances,  and  influences  that  affect  employment  of  military 

1.1.1.3.2.2 

Threat  Summary 

System  Threat  Assessment  Report  (STAR) 

The  sum  of  the  potential  strengths,  capabilities,  and  strategic  objectives  of  any  adversary  that  can  li 

1.1.1.3.3 

Tasks 

Functional  Area  Analysis  (FAA);  Functional  Needs  Assessment  (FNA);  Opet 

A  clearly  defined  and  measurable  activity  accomplished  by  individuals  and  organizations.  (FM  7-0) 

1.1.1.3.3.1 

Conditions 

Formation  OMS/MP  (Collective Tasks,  Conditions,  Standards);  System  ON 

Those  variables  of  an  operational  environment  or  situation  in  which  a  unit,  system,  or  individual  is 

1.1.1.3.3.2 

Standards 

Formation  OMS  (Collective  Tasks,  Conditions,  Standards);  System  OMS  (S 

A  quantitative  or  qualitative  measure  and  criterion  for  specifying  the  levelsof  performance  of  a  tasi 

1.1.1.3.3.3 

Measures  of  Performance  (MoP) 

Formation  OMS  (Collective Tasks,  Conditions,  Standards);  System  OMS  (S 

A  criterion  used  to  assess  friendly  actions  that  are  tied  to  measuring  task  accomplishment.  (JP  3-0) 

1.1.1.3.4 

Timeframe  and  Justification 

Required  Capabilities  (RC)  (published  byARCIC  and/or  COEs);  Army  Warf 

The  timeframe  considered  in  the  CBA  is  important  both  to  help  establish  the  conditions  and  threats  i 

1.1.1.3.5 

IDefense  Planning  Scenarios 

DI-IPSC-81431A/SEBOK 

This  is  a  graphic  and  narrative  description  of  area,  environment,  means  (political,  economic,  social. 

1.1.1.3.6 

Using  Organization(s)  (supported  SoS) 

Basis  of  Issue  (BOI)  Guidance 

1.1.1.3.6.1 

Quantities  issued  per  using  organization 

Basis  of  Issue  (BOI)  Guidance 

1.1.1.3.7 

Critical  Operational  Issues  and  Criteria  (COICs) 

Test  and  Evaluation  Master  Plan  (TEMP) 

1.1.1.4 

Potential  Non-Materiel  Solutions 

ICD 

These  are  changes  in  doctrine,  organization,  training,  materiel,  leadership  and  education,  personne 

1.1.1.5 

Materiel  Approaches 

ICD 

Correction  of  a  deficiency,  satisfaction  of  a  capability  gap,  or  incorporation  of  new  technology  that 

Challenges 

•Extensive  and  complex 
view  (The  Excel  file 
expands  to  over  400 
line  items) 

•Difficulty  discerning 
hierarchical  relationship 
between  data  elements 

•Very  manual  process  to 
render  diagrams  and 
show  relationships 
between  elements. 

•Cumbersome  to  track 
changes 
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DSM  Data  Taxonomy  in 
The  Brain  Mind  Mapping  Tooi 


Challenges 
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Modeling  the  DSM  Data  Taxonomy 


The  model  is  used  to  create  a  hierarchy  diagram  view. 


op  Level  Data  Taxonomy  Diagr 


□■"S  Digital  System  Model  (DSM) 

Relations 

Ell  Data  Taxonomy 
iS  "^  Relations 
S  -Ell  Diagrams 

]  Data  Taxonomy  Table 

i  m 

□■■■CH  1  System  Acquisition  Info 

&-IIII  1.1  Definition  Info  .... 

[j-OI  1.1.1  Requiremeriisjnfo 
[jl  -OI  1.1.2  Design  Info 
iS  "  d  1.1.3  Manufacturing^Jnfo 
[j  -OI  1.1.4  Verification  and  V^lida 
B"  Id  1.2  Product  Support  Info  . . 

[j  -OI  1.2. 1  Lifecyde  Sustainmenf^^ 

[j  -OI  1.2. 2  Technical  Managemen'  \ 
[j  -OI  1.2.3  Infrastructure  Managt 
B  -d  1.3  Management  Info 

j . Management  Info  .... 

[j]  -  d  1.3.1  Schedule  Info 
[j]-d  1.3.2  WBS  Info 

j . d  1.3.3  Finandal  Info 

[j]  "d  1.3.4  Risks^  Issues^  and  0p| 

[j  -  d  1.3. 5  Technical  Info 
■  Ell  DataHub_Data_Digii3l  System  Model  v2- 
B  -EII  Document  Taxonomy 
[j}  -Ell  Model  Taxonomy 
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Document  Taxonomy 


Digital  System  Model  {DSM)  a 


Definition  Info 
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*  ■  ■  "►!  System  Acquisition 
Info 


ld  =  "1" 


Requirements  Info 


iq=  "1.1.1" 


rh 


^  Design  Info 


Id  =  "1.1.2" 


rli 


Manufacturing  Info 


Id  =  "1.1.3" 


rfi 


Verification  and  Validation  Info 


Id  =  "1.1.4- 


Product  Support  Info 


Id  =  "1.2" 


Package  Elements 
establish  an  initial 
organizing  structure  for 

Model  Taxonoi  the  DSM  Taxonomy. 


J 


Requirement  elements 
capture  data  within 
information  categories 
in  the  DSM  Taxonomy. 


Management  Info 


Id  =  "1.3" 


rfi 


Lifecycle  Sustainment 

Management  Info 

Id  =  "1.2.1" 

rh 

Technical  Management  Info 

Id  =  "1.2.2" 

rfi 

Infrastructure  Management 

Info 

Id  =  "1.2.3" 

rli 

Schedule  Info 


Id  =  "1.3.1" 


WBS  Info 


Id  =  "1.3.2" 


Financial  Info 


Id  =  "1.3.3" 


Risks,  Issues,  and 
Opportunities  Info 


Id  =  "1.3.4" 


Technical  Info 

ld  =  "1 

1.3.5" 
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Modeling  the  DSM  Data  Taxonomy 

(cont.) 


•  The  model  is  used  to  create  a  table  View. 


# 

A  Id 

Name 

Source 

Text 

1 

1 

d]  System  Acquisition  Info 

This  taxonomy  represents  current  knowledge  about  data  classes  and  data  types  captured  in 
todays  defense  acquisition  systems  programs.  This  taxonomy  was  built  as  an  organizing 
construct  that  could  be  used  by  programs  as  an  aid  to  managing  their  data  and  defining 
viewpoints  that  would  need  to  be  auto  generated  from  the  Digital  System  Model. 

2 

1.1 

m  Definition  Info 

ISO 

A  requirement  is  a  statement  that  identifies  a  product  or  processes  operational^  functional^  or 
design  characteristic  or  constraint^  which  is  unambiguous^  testable^  or  measurable  and 
necessary  for  product  or  process  acceptability  (ISO  2007). 

3 

1.1.1 

m  Requirements  Info 

ISO 

A  requirement  is  a  statement  that  identifies  a  product  or  processes  operational,  functional,  or 
design  characteristic  or  constraint,  which  is  unambiguous,  testable,  or  measurable  and 
necessary  for  product  or  process  acceptability  (ISO  2007). 

4 

1.1. 1.4 

in  Customer/Stakeholder  Ne 

0I-IPSC-81431A/S 

Set  of  stakfiholder  requirements  are  clarified  and  translated  from  statements  of  need  into 
engineering-oriented  language  in  order  to  enable  proper  architecture  definition,  design,  and 
verification  activities  that  are  needed  as  the  basis  for  system  requirement  analysis. 

Stakeholder  needs  and  requirements  represent  the  views  of  those  at  the  business  or  enterprise 
operations  level— that  is,  of  users,  acquirers,  customers,  and  other  stakeholders  as  they  relate 
to  the  problem  (or  opportunity),  as  a  set  of  requirements  for  a  solution  that  can  provide  the 
services  needed  by  the  stakeholders  in  a  defined  environment.  Using  enterprise-level  life  cycle 
concepts  (see  Business  or  Mission  Analysis  for  details)  as  guidance,  stakeholders  are  led 
through  a  struclured  process  to  elicit  stakeholder  needs  (in  the  form  of  a  refined  set  of 
system-level  life-cycle  concepta).  Stakeholder  needs  are  transformed  into  a  defined  set  of 
Stakeholder  Requirement,  which  may  be  documented  in  the  form  of  a  model,  a  document 
containing  texlual  requirement  statement  or  both. 

5 

1.1. 1.4.4 

m  Capability 

ICD 

A  capability  is  the  ability  to  achieve  a  desired  effect  under  specified  standards  and  conditions 
through  combinations  of  means  and  ways  to  perform  a  set  of  tasks.  {TRADOC  Regulation 

71-20) 

6 

1.1.1. 4.4.4 

dl  Capability  Gap 

ICD 

The  inability  to  execute  a  spedfied  course  of  action.  The  gap  may  be  the  result  of  no  existing 
capability,  lack  of  profidency  or  suffidency  in  an  existing  capability  solution,  or  the  need  to 

rQi-.l-srQ  -sr.  QTU-in-Hi-.n  r-ar.-al-hili-K^  nr^li  i+irm-.  -h-h  -s  i+i  ira  rrsr.  Qoo  1  “TH  Hi 

20^^  NDIA  SE  Conference 
Oct  25,  2017  1  Page-10 


Distribution  Statement  A  -  Approved  for  public  release  by  DOPSR  on  10/03/2017,  SR  Case  #  18-S-0007  applies.  Distribution  is  unlimited. 


Modeling  the  DSM  Data  Taxonomy 
(Data  Field  Descriptions) 


*  is  the  number  of  the  data  element. 

*  “ID”  indicates  the  hierarchical  location  of  the  data 
element  in  the  Data  Taxonomy. 

*  “Name”  provides  a  unique  name  for  each  data 
element  In  the  Data  Taxonomy. 

*  “Source”  provides  one  or  more  references  that  were 
used  to  derive  the  data  element. 

*  “Text”  provides  a  definition  for  each  data  element. 
Use  this  column  to  understand  what  data  to 
captured  for  each  of  the  associated  data  elements. 
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Benefits  to  Modeling  the 
DSM  Data  Taxonomy 


*  Manage  Complexity 

-  Provides  a  method  to  use  and  navigate  the  DSM  Data 
Taxonomy 

-  Manages  hierarchicai  data  structure 

*  Preserve  and  Enable  Reuse  of  Heritage  Knowiedge 

-  Provides  a  method  to  capture,  store,  and  use/reuse  data 

-  Offers  accessible,  shareable,  and  transparent  data  for  current 
and  future  workforce 

*  Outline  Data  Structure 

-  Provide  an  organized  structure  for  the  types  of  program  data 
that  should  be  considered  across  the  life  cycle 
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Path  Forward 


*  Content  Validation  of  DSM  Data  Taxonomy 

-  Work  with  Services  to  review  and  provide  comment  on  the  DSM 
Data  Taxonomy 

-  Incorporate  into  INCOSE  Digital  Artifact  Challenge 

*  Finalize  and  deploy  DSM  Data  Taxonomy  for  Usage 
after  Reviews  and  Revisions 

*  Model  Document  and  Model  Taxonomies 

*  Manage  Changes 
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Systems  Engineering: 
Criticai  to  Defense  Acquisition 


Defense  Innovation  Marketplace 

http:/ /WWW.  defenseinno  vationmarketplace.  mil 


DASD,  Systems  Engineering 

http://www.acq.osd.mil/se 
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For  Additional  Information 


Philomena  Zimmerman 
ODASD,  Systems  Engineering 

571-372-6695  |  philomena.m.zimmerman.civ@mail.mil 

Other  Contributors: 

Frank  Saivatore 

973-265-9837  |  frank.j.saivatore.ctr@mail.mil 

Tracee  Waiker  Gilbert,  Ph.D. 

571-372-6145  |  tracee.w.gilbert.ctr@maii.mii 

Tyesia  Pompey  Alexander,  Ph.D. 

571-372-6697  |  tyesia.p.alexander.ctr@mail.mil 

Allen  Wong 

571-372-6788  |  allen.wong4.ctr@mail.mil 
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DoD  Joint  Federated  Assurance  Center  (JFAC) 

2017  Update 

Thomas  Hurt 

Office  of  the  Deputy  Assistant  Secretary  of  Defense 

for  Systems  Engineering 

20th  Annuai  NDiA  Systems  Engineering  Conference 
Springfield,  VA  |  October  26,  2017 
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How  Did  We  Get  Here? 


LEGEND 


Policy  &  Guidance  H  Congressional  Actions 


2012 


2017 


Two  Questions  for  the 
Record 


Enclosure  14  - 
Cybersecurity 
Do  Dl  5000.02 


DoD  Software  Assurance 
(SwA)  Tiger  Team 


2011 

DoD  SwA  Strategy 
FY1 1  NDAA,  Sec.  932 


2014 

Establish  JFAC 
FY14  NDAA.  Sec.  937 

2013 


2017 


JFAC  SwA  Capability 
Gap  Analysis 


SwA  Automation 
FY13  NDAA,  Sec.  933 


DSB  Task  Force  on 
Cyber  Supply  Chain 


Congress  and  DoD  have  acknowledged  the  need  for  increased  software 
assurance  to  improve  confidence  in  secure  and  resilient  weapon  systems 

for  over  a  decade. 


JFAC:  Joint  Federated  Assurance  Center 
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Joint  Federated  Assurance  Center 

(JFAC) 


FY14  NDAA  Section  937 — Joint  Federated  Assurance  Center  (JFAC) 
Key  provisions:  Charter  elements: 


“provide  for  the  establishment  of  a 
joint  federation  of  capabilities  to 

support  the  trusted  defense  system 
needs... to  ensure  security  in  the 
software  and  hardware  developed, 
acquired,  maintained,  and  used  by  the 
Department” 


-  Role  of  federation  in  supporting 
program  offices 

-  SwA  and  HwA  expertise  and 
capabilities  of  the  Federation,  including 
policies,  standards,  requirements,  best 
practices  contracting,  training  and 
testing 


-  “consider  whether  capabilities  can  be 
met  by  existing  centers” 


R&D  program  to  improve  code 
vulnerability  analysis  and  testing  tools 


“[if  gaps]  shall  devise  a  strategy 
[for]  resources  [to  fill  such 
gaps]” 


Requirements  to  procure  manage,  and 
distribute  enterprise  licenses  for 
analysis  tools 


-  “[NLT  180  days,  SECDEF  shall]  issue 

a  charter...” 


-  “submit  to  congressional  defense 
committees... a  report  on  funding  and 
management” 
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What  Has  DoD  Done? 


•  Development  of  Concept  of  Operations  (CONOPs)  and  Charter 

•  Establishment  of  JFAC  Coordination  Center  (JFAC-CC), 
Steering  Committees,  Working  Groups  (WGs) 

•  Piloting  Software  Assurance  (SwA)  license  distribution  and 
management 

•  Conduct  SwA  and  Hardware  Assurance  (HwA)  Capability  Gap 
Anaiysis 
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JFAC  Operational  Structure 


DepSecDef 


USD(AT&L) 


AT&L  CIO 

Army  DISA 

Navy  NSA 

Air  Force  NRO 
MDA  DMEA 

DOE  • 

Policy  and  Technical 
AOs  assigned  by 
above  organizations 


SwA  and  HwA  Working  Groups 

-  Collaboration  and  shared  prioritization  in 
daily/weekly  activities,  meet  on  a  regular  basis 

-  Recommend  policy  and  guidance 

-  Provide  community  forum  for  “hard  problem’ 
analysis  and  question/answer 

JFAC  Coordination  Center 

-  Coordination  of  Service  Providers 

-  Supports  programs  with  situational  awareness! 
information/best  practices,  coordination 

-  SwA  analysis  tool  license  distribution 

-  Portal:  https://ifac.armv.mii 

-  Assessment  Knowledge  Base  (future) 

JFAC  Action  Officer  (AO)  WG 

-  AOs  for  JFAC  Steering  Committee 

-  Maintain  enterprise  and  strategy  cognizance 

-  Reporting  and  ROI  status 
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What’s  Going  On  Now? 


•  JFAC  Web  portal  and  SwA  tool  license  distribution 

•  Security  Classification  Guide 

•  Field  Programmable  Gate  Array  (FPGA)  Strategy 

•  Resourcing 
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What’s  Next? 


•  Develop  JFAC  Full  Operational  Capability  (FOC)  strategy 

-  Improve  DoD  SwA  throughout  Lifecycle  Planning,  Execution  and  Sustainment 

-  Invest  in  Technology  and  Resources 

-  Upgraded  Infrastructure  for  Federated  DoD-wide  Coordination  of  Software 
Assurance 

-  Linking  Sustainment  to  Early  Program  Development 


•  JFAC  website  on  SIPR,  JWICS 

-  One-stop  shop  for  SwA  tools  and  best  practices 

-  New  S&T  and  Assessment  Knowledge  Base  portals 

-  https://jfac.army.mil 
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Conclusion 


•  The  JFAC’s  goal  is  to  provide  DoD  programs  a  one-stop  shop 
to  request,  evaluate,  and  obtain  resources  to  improve  their 
software  assurance  practice. 

-  SwA  analysis  tool  license  distribution  and  management 

-  Service  providers  for  programs’  SwA  work;  SMEs  focused  on  hard  problems 

-  SwA  best  practices 

•  JFAC  is  addressing  key  software  assurance  gaps. 

-  Developing  FOC  strategy  to  execute  as  resourcing  becomes  available 

-  Publishing  best  practices  at  JFAC  web  portal  (https://jfac.army.mil) 
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Systems  Engineering: 
Criticai  to  Defense  Acquisition 


Defense  Innovation  Marketplace 

http:/ /WWW.  defenseinno  vationmarketplace.  mil 


DASD,  Systems  Engineering 

http://www.acq.osd.mil/se 
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For  Additional  Information 


Mr.  Thomas  Hurt 
ODASD,  Systems  Engineering 

571  -372-61 29 

thomas.d.hurt.civ(g)mail.mil 
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Achieving  DoD  Software  Assurance  (SwA) 


Thomas  Hurt 

Office  of  the  Deputy  Assistant  Secretary  of  Defense 

for  Systems  Engineering 

20th  Annual  NDIA  Systems  Engineering  Conference 
Springfield,  VA  |  October  26,  2017 
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First  Line  of  Defense  in  Software  Assurance  is 
the  Application  (Software)  Layer 


Software  assurance 
(SwA)  provides  the 
required  level  of 
confidence  that  software 
functions  as  intended 
(and  only  as  intended) 
and  is  free  of  (known) 
vulnerabilities,  either 
intentionally  or 
unintentionally  designed 
or  inserted  in  software, 
throughout  the  life 
cycle. 


84%  of  breaches  exploit 
vulnerabilities  in  the 
appiication^ 

Yet  funding  for  IT  defense  vs. 
software  assurance  is  23  to  1^ 


1.  Clark,  Tim,  “Most  Cyber  Attacks  Occur  from  This 
Common  Vulnerability,”  Forbes,  03-10-2015 

2.  Feiman,  Joseph,  “Maverick  Research:  Stop 
Protecting  Your  Apps;  It’s  Time  for  Apps  to 
Protect  Themselves,”  Gartner,  09-25-2014. 


G00269825 
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How  Did  We  Get  Here? 


LEGEND 


Policy  &  Guidance  H  Congressional  Actions 


2012 

Two  Questions  for  the 
Record 


2017 

Enclosure  14  - 
Cybersecurity 
DoD 1 5000.02 


2004  -  2006 

DoD  Software  Assurance 
(SwA)  Tiger  Team 


2011 

DoD  SwA  Strategy 
FY1 1  NDAA,  Sec.  932 


2014 

Establish  JFAC 
FY14  NDAA.  Sec.  937 

2013 

SwA  Automation 
FY13  NDAA,  Sec.  933 


2017 

JFAC  SwA  Capability 
Gap  Analysis 

DSB  Task  Force  on 
Cyber  Supply  Chain 


Congress  and  DoD  have  acknowledged  the  need  for  increased  software 
assurance  to  improve  confidence  in  secure  and  resilient  weapon  systems  for 

over  a  decade. 


JFAC:  Joint  Federated  Assurance  Center 
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How  to  Engineer  Software  Assurance 
Across  the  DoD  Acquisition  Life  Cycle 
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SwA  within  DoD 


DepSecDef 


USD(AT&L) 


AT&L 


CIO 


Army  DISA 

Navy  NSA 

Air  Force  NRO 
MDA  DMEA 

DOE 

Policy  and  Technical 
AOs  assigned  by 
above  organizations 

_ 


JFAC  SwA  Working  Group 

-  Collaboration  and  shared 
prioritization  in  daily/weekly 
activities,  meet  on  a  regular  basis 

-  Recommend  SwA  policy  and 
guidance 

-  Provide  community  forum  for  “hard 
problem”  analysis  and 
question/answer 


DoD  SwA  Community  of 

Practice 

-  Tri-leads;  meets  quarterly  with 
various  DoD  stakeholders’ 
participation 

-  Sponsors  research  and  pilots  into 
hard  SwA  problems 
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What’s  Going  on  Now?  (1  of  3) 


•  DoD  Software  Assurance  Community  of  Practice 

-  Past  products  include:  Contract  language  for  integrating  SwA;  State-of-the-Art 
Resource  (SOAR)  for  SW  Vulnerability  Detection,  Test,  and  Evaluation;  SwA 
metrics 

-  Recent  Topics  and  Ongoing  Activities 

o  SwA  Risk  Assessment  process 
o  Malware  discovery  in  binary  code 
o  SwA  analysis  of  mobile  software 

*  The  Journal  of  Cyber  Security  and  Information 
Systems:  Design  &  Development  Process  for  Assured 
Software-Vol  1  * 

-  Software  Assurance  in  the  Agile  Software  Development  Lifecycle 

-  Is  Our  Software  REALLY  Secure? 

-  Development  and  Transition  of  the  SEI  Software  Assurance  Curriculum 

-  Keys  to  Successful  DoD  Software  Project  Execution 

-  Hacker  101  &  Secure  Coding:  A  Grassroots  Movement  toward 
Software  Assurance 


*  https://www.csiac.org/journal-issue/design-and-development-process-for-assured-software-volume-1/ 
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What’s  Going  on  Now?  (2  of  3) 


> 

1 

Design 

iS 

Inspection 

_ > 

_ / 

Secure 

Coding 

Practices 


ents 

ty 


Developmental 
Release  Decisiol 


Acquisition  Phase 
Considerations 


Systems  Engineering  Technical 
Review  Success  Criteria 


SOFTWARE  ASSURANCE  CONSIDERATIONS  (TMRR  Phase) 


A 


irporate  SwA  requirements,  tool  use,  metrics,  and  assurance  thresholds  into  solicitations.  Architectures, 
'igns,  and  code  developed  for  prototyping  are  frequently  reused  later  in  development. 

■■■■-..■isess  system  functional  requirements  and  verification  methods  for  inclusion  of  SwA  tooli 
remediation  across  the  development  life  cycle. 

.sess  requirements  for  SwA  are  correct  and  complete  regarding  assurance.  Consider  mea 


SwA  Success  Criteria 


Technology 
Maturation  & 
Risk  Reduction 


^  ^  satisfies  user  requirements  and  developer  ready  to 

fcrs  and  adversaries  using  malicious  inserts;  system  characteristics;  interoperability  wn  .  .  i 

®  ^  f  j  begin  detailed  design  with  acceptable  risk, 

lion  threads;  and  other  factors.  Assure  that  mapping  and  traceability  are  maintained  as  : 


■.stream  assessments. 


blish  baseline  architecture  and  review  for  weaknesses  (e.g.,  use  of  Common  Weakness  design  provides  sufficient  confidence  that  the 


SR  SRR  SFR  PDR 


Preliminary  design  and  basic  system  architecture 


I))  and  susceptibility  to  attack  (e.g.,  use  of  Common  Attack  Pattern  Enumeration  and 
iC)),  and  likelihood  of  attack  success  considering  each  detected  weakness;  identify  p 
points  and  mission  impacts.  Consider  which  families  of  automated  SwA  engineering  tools 
vulnerability  or  weakness  detection. 

•  Review  architecture  and  design  for  adherence  to  secure  design  principles  and  assess  sound)  support  capability  need  and  affordability  target 
decisions  considering  likely  means  of  attack;  programming  language  choices;  developmeni  achievement, 
frameworks;  and  use  of  open  source  software,  etc. 

*  Identify  and  mitigate  technical  risks  through  competitive  prototyping  while  engineering  in 
prototypes  may  be  physical  or  math  models  and  simulations  that  emulate  expected  perform 
concepts  may  require  scaled  models  to  reduce  uncertainty  too  difficult  to  resolve  purely  by 
emulation.  SW  prototypes  that  reflect  the  results  of  key  trade-off  analyses  should  be  demoi 
TMRR  phase.  These  demonstrations  will  provide  SW  performance  data  (e.g.,  latency,  seci 
integration  of  legacy  services,  graceful  function  degradation  and  re -initiation,  and  scalabili 
decisions  as  to  maturity;  further,  EMD  estimates  (schedule  and  life  cycle  cost)  often  depen 
components  developed  in  TMRR;  therefore  to  prevent  technical  debt,  SwA  considerations  must  have  been  taken 
into  account. 

>  Develop  a  comprehensive  system-level  architecture,  then  design  (address  function  integrity,  assurance  of  the 
functional  breakout,  function  interoperation,  and  separation  of  function)  that  covers  the  full  scope  of  the  system 
in  order  to  maintain  capabilities  across  multiple  releases  and  provide  the  fundamental  basis  to  fight  through 


Recommendation  that  allocated  baseline  fully 


Preliminary  Design  Review  (PDR) 


Allocated  baseline  is  established  such  that  the 


program  demonstrates  a  high  likelihood  of 
accomplishing  its  intended  mission,  including  in  a 
cyber-contested  environment. 


•  Determine  that  baseline  fully  satisfies  user  requirements, 
with  assurance  engineered  in. 

•  Determine  that  likely  means  of  attack  through  software 
have  been  assessed  and  used  in  architecture  and  design 
implementation. 

•  Review  architecture  and  design  against  secure  design 
principles;  including  system  element  isolation,  least 
common  mechanism,  least  privilege,  fault  isolation,  input 
checking  and  validation.  Consult  JFAC  planning  tools, 
best  practices  in  architecture  and  design,  and  guidance. 

•  Determine  if  initial  SwA  Reviews  and  Inspections  from 
prior  SETR  activities  capture  planning  and  requirements 
appropriately,  including  assurance. 

•  Confirm  that  SwA  requirements  that  were  previously 
mapped  from  tactical  use  threads,  mission  threads, 
system  requirements,  and  system  interoperability 
requirements,  are  mapped  to  module  test  cases  and  to  the 
final  acceptance  test  cases. 

•  Establish  automated  regression  testing  procedures  and 
tools  as  a  core  process,  and  assure  regression  testing  is 
conducted  for  remediated  vulnerabilities,  defects,  and 
weaknesses. 


PM’s  Guidebook 
for  SwA  Activities 


4 


DoD  Program  Manager's  Ciuidebuok  for 
Integrating  Sullwarc  Assurance 
Engineering  Activities  into  the  System 
Acquisition  Lifecycle 


To  be  published  by  SEI. 


Upcoming  Journal  of  Cyber  Security  and  Information  Systems  article: 
“Engineering  SwA  into  Weapon  Systems  during  the  DoD  Acquisition  Life  Cycle’ 
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What’s  Going  on  Now?  (3  of  3) 


In  July  2016,  the  JFAC  SwA  Technical  Working  Group  identified  63  DoD 
capability  gaps  that  prevent  the  effective  planning  and  execution  of  software 
assurance  within  the  DoD  acquisition  process.  The  gaps  were  organized  into 
seven  categories: 


Gap  Examples: 

2.2.2  -  SwA  requirements  lacking  in  system 
requirements 

5.2.1  -  Lack  of  SwA  training  for  Program  Managers 

6.1  -  Lack  of  definitive  contract  language  for  SwA 
planning  and  execution  activities,  as  early  in  the 
lifecycle  as  possible 


■  Lifecycle  Planning  and  Execution 

■  Technology 

■  Policy,  Guidance,  and  Processes 

■  Resources 

■  Contracting  and  Legal 

■  Metrics 

■  Federated  Coordination 


As  chair  of  the  JFAC  Steering  Committee,  Ms.  Kristen  Baldwin,  Acting  Deputy 
Assistant  Secretary  of  Defense  for  Systems  Engineering  (DASD(SE)),  approved 
the  analysis*  and  directed  the  Technical  Working  Group  to  develop  a  strategy  to 
address  the  identified  gaps.  DASD(SE)’s  JFAC  lead,  Mr.  Tom  Hurt,  supported 

the  NDIA-sponsored  joint  industry-government  workshop. 


*Distribution  C,  available  upon  request. 
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What’s  Next? 


•  DoD  Program  Manager’s  Guidebook  for  Integrating  Software 
Assurance  Engineering  Activities  into  the  System  Acquisition 
Life  Cycle 

-  To  be  written  and  published  by  SEI  in  collaboration  with  JFAC  SwA  Technical  WG 

-  Partner  Document:  Software  Developers  Guidebook 

•  DASD(SE)  Activities 

-  FY18  Business  Case  Analysis  for  SwA  Tools 

•  JFAC  website  on  SIPR,JWICS 

-  One-stop  shop  for  SwA  tools  and  best  practices 

-  New  S&T  and  Assessment  Knowledge  Base  portals 

-  https://jfac.army.mil 

•  Develop  JFAC  Full  Operational  Capability 
(FOC)  strategy 

-  Improve  DoD  SwA  throughout  Lifecycle  Planning,  Execution  and  Sustainment 

-  Linking  Sustainment  to  Early  Program  Development 


DoD  Program  Manager’s  Guidebook 
for  Ensuring  Software  Assurance 
Activities  are  Engineered-ln 
throughout  the  System  Acquisition 
Lifecycle:  Draft  Project  Plan, 
Version  0.7 


SwA  COP  Workshop 
July  27-28,  2016 
Institute  for  Defense  Analyses 
Alexandria,  Virginia 


Soflwar*  Engin««rlng  insMute 
Canvegie  Mellon  University 
Pittsbirgh.  PA  15213 


r  Software  Engineering  Institute  |  (4ini{%'leMrlhMil  nlu‘rHti> 
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Conclusion 


•  DoD  has  been  focused  on  software  assurance  for  over  a  dozen 
years. 

-  DASD(SE)  leads  the  development  and  implementation  of  the  supporting  best 
practices,  guidance,  tools,  and  workforce  competencies  to  ensure  PMs  have  the 
means  to  mitigate  SwA  vulnerabilities  and  risk. 


•  The  JFAC’s  goal  is  to  provide  DoD  programs  a  one-stop  shop 
to  request,  evaiuate,  and  obtain  resources  to  improve  their 
software  assurance  practice. 

-  SwA  analysis  tool  license  distribution  and  management 

-  Service  providers  for  programs’  SwA  work;  SMEs  focused  on  hard  problems 

-  SwA  best  practices 

•  JFAC  and  DoD  SwA  COP  is  addressing  key  software  assurance 
gaps. 

-  Developing  FOC  strategy  to  execute  as  resourcing  becomes  available 
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Systems  Engineering: 
Criticai  to  Defense  Acquisition 


Defense  Innovation  Marketplace 

http:/ /WWW.  defenseinno  vationmarketplace.  mil 


DASD,  Systems  Engineering 

http://www.acq.osd.mil/se 
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For  Additional  Information 


Mr.  Thomas  Hurt 
ODASD,  Systems  Engineering 

571  -372-61 29 

thomas.d.hurt.civ(g)mail.mil 
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Digital  Engineering  and 
Engineered  Resilient  Systems  (ERS) 


Mr.  Robert  Gold 
Director,  Engineering  Enterprise 
Office  of  the  Deputy  Assistant  Secretary  of  Defense 

for  Systems  Engineering 

20th  Annuai  NDiA  Systems  Engineering  Conference 
Springfieid,  VA  |  October  26,  2017 
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Industrial 

Revolution 


MECHANICAL 

Use  of  mechanical 
production  powered  by 
water  and  steam 


History 


2"^  Industrial 
Revolution 


ELECTRICAL 

Use  of  mass  production 
powered  by  electrical 
energy 


3*'^  Industrial 
Revolution 


INFORMATION 

TECHNOLOGY 

Use  of  electronics  and 
IT  to  further  automation 


4th  Industrial 
Revolution 


DIGITAL 

Use  of  a  digitally 
connected  end-to-end 
enterprise 


I 

1800 


I  I  I  I  I  I  I 

1900 

Traditional  Models  and 
Simulations  (M&S) 


Simulation  Based  Acquisition  (SB A) 


T 


T 


2000  TODAY 

Model-  1  DIGITAL 
Based  ENGINEERING 

Systems  (DE) 

Engineering 
(MBSE) 
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Digital  Engineering: 
MBSE  approach  for  DoD 


Current  State 

■  Our  workforce  uses  stove-piped  data  sources  and  models  in  isolation  to  support  various 
activities  throughout  the  life-cycle 

■  Current  practice  relies  on  standalone  (discipline-specific)  models 

■  Communication  is  through  static  disconnected  documents  and  subject  to  interpretation 


Future  State 


■  Digital  Engineering  moves  the  engineering  discipline  towards  an  integrated  model-based 
approach 

■  Through  the  use  of  digital  environments,  processes,  methods,  tools,  and  digital  artifacts 

■  To  support  planning,  requirements,  design,  analysis,  verification,  validation,  operation, 
and/or  sustainment  of  a  system 

■  Digital  Engineering  ecosystem  links  our  data  sources  and  models  across  the  lifecycle 

■  Provides  the  authoritative  source  of  truth 


iReQC/zre/n  Gn  ts 


Assemblies 


Subsystems 

mi 


Others 


IVIaterial 

Solution 

Analysis 


Technology 
Maturation  and 
Risk  Reduction 


Engineering  and 
Manufacturing 
Development 


Production  and 
Deployment 


Operations 
and  Support 


Manufacturing 

Systems 


Configuration  Test 

Management 


Current:  Stove-piped  models  and  data  sources  Future:  Digital  Engineering  Ecosystem 
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ERS  Products  in  Digital 
Engineering  Context 


Digital  Engineering 

*  Digital  Engineering  vision 
moves  the  engineering 
discipline  towards  an  integrated 
model-based  approach  through 
the  use  of  digital  environments, 
processes,  methods,  tools,  and 
digital  artifacts 

•  Model  is  a  representation  of 
reality 

-  Model  is  ‘composed  of  data,  algorithms 
and/or  processes 

-  Computable  or  used  in  a  computation 


ERS 


•  Engineered  Resilient  Systems 
(ERS)  combines  advanced 
engineering  techniques  with 
high-performance  computing  to 
develop  concepts  and  tools  that 
significantly  amplify  design 
options  examined 

•  Develop/Integrate  advanced 
engineering  tools  for  efficient, 
integrated  design  and 
development  across  the  full 
range  of  the  product  lifecycle 
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Digital  Engineering  Relationships 


\ 


Supporting  tools: 
(Large  Tradespace 
Analytics  datasets, 
Analysis  of 
Alternatives,  Virtual 
Prototyping 
Evaluation,  etc.) 

INGWEEeRESlIEIfl  SYSTEMS 


User  selected  and  integrated  based  on  outcome  needed 

Traditional  Other  Physics-based  / 

Mod/Sim  Initiatives  Engineering 

Solutions  Design  Tools 


^CREATE 


Simulation  Coordination 
Office  (DMSCO) 


Computational  Research  and 
Engineering  Acquisition  Tools  and 
Environments  (CREATE) 


World-class 
Computational 
Resources  (High 
Performance 
Computing),  Software, 
Networking 

DOD 

HEC 

MODERNIZATION  PROGRAM 
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Transitioning  S&T  to  Engineering  & 

Acquisition 


T 


Valley  of  Death  Vi 


-  A  ^ 


T 


Historical  User  Community 


Target/Expanded  User  Community 
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DRAFT  Vision  for  ERS,  CREATE,  et  al 
(crossing  the  Vailey  of  Death) 


DRAFT 


DRAFT 


Current  Domains:  Air  (Fixed  &  Rotary),  Surface,  Subsurface,  Ground,  RF,  Meshing,  Geometry 
Future  Domains:  Space,  Flypersonics,  Improved  Turbine  Engine,  EW,  Directed  Energy,  Others? 


/bV 

/K 

IOC 

FOC 

JCIDS  —  ICD,  CDD,  CPD  Materiel  Solution 

Technology  Maturation 

Engineering  &  Manufacturing 

Production  and 

Operations  and 

AoA  -  Guidance/Plan  Analysis 

&  Risk  Reduction 

Development 

Deployment 

Support 

Current  ERS  Uses 

Future  ERS  Uses 

1 

Future  CREATE  Uses 


EC&P  use  of  ERS,  CREATE  and  other  tools  and  environments 


Proof  of  Principle  Prototypes 


Fieldable  Prototypes 


Pre-EMD  Prototypes 


Current  =  []  B  Future  = 


DT&E  use  of  ERS,  CREATE  other  tools  and  environments 


Other 


Future  ERS  Use:  Industry 


I  Force  Effectiveness/Mission  models 


Force  Eff  /  Msn  Models 


Engineering  Models 


I  Eng  Models 


System  CONORS 


System  CONORS 


Digital  System  Model  /  Digital  Thread 


1  Digital  Twin 

1  CAD  /  CAM  /  Add  Mfg  | 
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Digital  Engineering  Strategy: 

Five  Goals 


Formalize  the  development,  integration 

and  use  of  models  to  inform  enterprise  and 
program  decision  making 


Provide  an  enduring  authoritative  source 
of  truth 


Incorporate  technological  innovation  to 

improve  the  engineering  practice 


Establish  supporting  infrastructure  and 
environments  to  perform  activities,  collaborate, 
and  communicate  across  stakeholders 


Transform  a  culture  and  workforce  that  adopts 
and  supports  Digital  Engineering  across  the  lifecycle 


Drives  the  engineering  practice  towards  improved  agility,  quality,  and 
efficiency,  resulting  in  improvements  in  acquisition 
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Goal  #1 :  Formalize  Development, 
Integration  &  Use  of  Models 


Specialty  Engineering  Models 


Product 

Support 

Models 


Management 

Models 


System 

Models 


I  Design 
y]  Models 


Verification  and 
Validation  Models 


Manufacturing 

Models 


Authoritative 
Source  of 
Truth 


Key:  Data 


ERS  in  DE  Goal  1 : 

•  Use  of  models  to  replace  the 
sequential,  fixed  requirement 
approach  to  design 

•  Use  of  models  will  enable 
prototyping,  experimenting  and 
testing  of  solutions  virtually 
before  physical  prototypes  and 
full  scale  systems  are  available 

•  Use  of  evolving  models  will  allow 
analysis  of  design  options  to  be 
shifted  left  in  the  lifecycle 

•  Understand  how  to  defeat  a 
concept  through  inverse 
modeling 


Models  as  the  cohesive  element  across  a  system’s  lifecycle 
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Goal  #2:  Provide  an  Authoritative 

Source  of  Truth 


Stakeholders 


•  Auditing 

•  Business  -  Cost  Estimating 

•  Business  -  Financial 
Management 

•  Contracting 

•  Engineering 

•  Facilities  Engineering 

•  Industrial  Contract  Property 
Management 

•  Information  Technology 

•  Life  Cycle  Logistics 

•  Production,  Quality,  & 
Manufacturing 

•  Program  Management 

•  Purchasing 

•  Science  &  Technology 
Management 

•  Test  and  Evaluation 


ERS  in  DE  Goal  2: 

•  Models  are  inherently  more  adaptable  across  mission  sets  and  environments 

•  The  authoritative  sources  of  truth  means  ground  truth 

•  ERS  is  fast  and  accurate  enough  to  understand  and  mitigate  risk  in  large,  complex, 
and  integrated  data  set 


Right  information,  right  people,  right  uses,  right  time 
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Goal  #3:  Incorporate 
Technological  Innovation 


Big  Data  and  Analytics 
Cognitive  Technologies 
❖  Computing  Technologies 
Digital-to-Physical  Fusion  Technologies 


ERS  in  DE  Goal  3: 

•  Explore  new  concepts  to  integrated 
advanced  engineering  models 

•  Replace  intensive  manual  processes 
to  stitch  data  and  artifacts  together 
with  workflow  automation 

•  Explore  new  decision  analytics  that 
generate  real  alternatives  that  reflect 
the  entire  lifecycle  demanded  by 
increased  digital  engineering  use 

•  Utilize  machine  learning  to  analyze 
massive  and  complex  datasets 
containing  a  variety  of  data  types 
from  a  multitude  of  sources 

•  Architecturally  integrated  with 
knowledge  management 


Harness  technology,  new  approaches,  and  human-machine 
collaboration  to  enable  an  end-to-end  digital  enterprise 
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Goal  #4:  Establish  Infrastructure  & 

Environments 


ERS  in  DE  Goal  4: 

•  Architect  an  overall  data  ecosystem  on  HPCs 

•  Build  generalized  and  reusable  workflow  engine 

•  Build  enterprise-level  web  portal 

•  Organize  software  tools  around  the  data 

•  Create  visualization  techniques  that  support  decision  makers 


Foundational  support  for  Digital  Engineering  environments 
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Goals  #5:  Transform 
Culture  and  Workforce 


Communication 
and  Engagement 


Leadership  Training  and 
Education 

Strategy  and  Continuous 

Implementation  Improvements 


Culture  and  Woi 
Enablers 


ERS  in  DE  Goal  5: 

•  Understand  that  migrating  to  a  digital 
ecosystem  does  not  remove  the 
responsibility  from  the  users  to  select, 
manage,  govern  and  use  the  tools 
appropriately 

•  Gain  confidence  in  performing  activities  in  a 
collaborative,  integrated,  digital  model-based 
environment 

•  Learn  to  articulate  the  problem,  workflow, 
and  model  boundary  conditions  to  a  third 
party 

•  Build  understanding  in  how  to  appropriately 
reduce  reliance  on  physical  experimentation 


Institutionalize  Digital  Engineering  across  the  acquisition  enterprise 


20^^  NDIA  SE  Conference 
Oct  26,  2017  1  Page-13 


Distribution  Statement  A  -  Approved  for  public  release  by  DOPSR  10/03/2017.  Case  #  18-S-0008  applies.  Distribution  is  unlimited. 


There  Is  Much  More  to  Do... 


•  Publish  the  Digital  Engineering  Strategy 

-  Support  development  of  implementation  guidance/direction  in  Services/Agencies 

-  Follow  with  policy? 

•  Finish  the  Digitai  Engineering  Starter  Kit 

-  Continue  development;  share/obtain  feedback  on  digital  artifact  use 

•  Engage  with  Acquisition  Programs 

-  Establish  criteria  for  use  of  Digital  Engineering  artifacts  for  decision  points 

•  Update  Competencies  across  Acquisition  Curricuia 

-  Identify  Digital  Engineering  education  and  training  outside  of  acquisition  curricula 

•  Update  Poiicy  and  Guidance  (Engineering,  et  ai) 

-  Develop/update  governance  processes,  policy,  guidance  and  contracting  language 

•  Transform  Acquisition  Practice 

-  Engage  acquisition  users 

-  Incorporate  rigor  from  Digital  Engineering  practices  and  artifacts  into  system  lifecycle 


activities 
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Systems  Engineering: 
Criticai  to  Defense  Acquisition 


Defense  Innovation  Marketplace 

http:/ /WWW.  defenseinno  vationmarketplace.  mil 


DASD,  Systems  Engineering 

http://www.acq.osd.mil/se 
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For  Additional  Information 


Mr.  Robert  Gold 
ODASD,  Systems  Engineering 

703-695-31 55 

robert.a.gold4.civ(g)mail.mil 
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Digital  Engineering  Overview 


*  Background 

-  Dynamic  operational  and  threat  environments 

-  Growth  in  system  complexity  and  risks 

-  Linear  acquisition  process  that  lacks  agility 
and  resiliency 

-  Cost  overruns  and  delayed  delivery  of  capabilities  to  the  warfighter 

-  Current  practices  can’t  keep  pace  with  innovation  and  technology 
advancements 

•  Need 

-  Outpace  rapidly  changing  threats  and  technological  advancements 

-  Deliver  advanced  capabilities  more  quickly  and  affordably  with 
improved  sustainability  to  the  warfighter 

-  Foster  a  culture  of  innovation 


Digital  Engineering:  An 
integrated  digital  approach  that 
uses  authoritative  sources  of 
systems'  data  and  models  as  a 
continuum  across  disciplines 
to  support  lifecycle  activities 
from  concept  through  disposal. 


Digital  Engineering  transforms  the  way  the  DoD  innovates  and  operates 
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Digital  Models  Have  Incredible 

Potential 


DoD  needs: 

•  Flexible  designs  that  adapt  and  are  resilient 
to  unknown  missions  and  threats 

•  Cost  and  affordability  as  quantifiable 
attributes  of  the  trade  space 

•  Systems  of  Systems,  and  Enterprise, 
contexts  in  order  to  respond  to  multiple 
stakeholders 

•  A  balance  between  agility  in  acquisition  and 
rigorous  analysis  and  data 

•  Critical  information  appropriately  protected 
while  designing  for  interoperability 

•  Support  in  significantly  diverse  domains 


Balancing  these  axioms 
is  challenging. 

It  drives  the  need  for, 
and  use  of  digital 
models  to: 

•Maintain  consistency 
about  the  system 

•Integrate  technical  and 
non-technical  drivers 

•  Understand  the 
various  perspectives 
on  the  system  under 
development 


Models  are  advancing  the  STATE  OF  PRACTICE  of  SE 
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Digital  Engineering  (DE)  and  Computational 
Research  and  Engineering  Acquisition  Tools 

and  Environments  (CREATE) 

Ms.  Phil  Zimmerman 

Deputy  Director,  Engineering  Toois  and  Environments 
Office  of  the  Deputy  Assistant  Secretary  of  Defense 

for  Systems  Engineering 

20th  Annual  NDIA  Systems  Engineering  Conference 
Springfieid,  VA  |  October  25,  2017 
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Industrial 

Revolution 


MECHANICAL 

Use  of  mechanical 
production  powered  by 
water  and  steam 


History 


2"^  Industrial 
Revolution 


ELECTRICAL 

Use  of  mass  production 
powered  by  electrical 
energy 


3*'^  Industrial 
Revolution 


INFORMATION 

TECHNOLOGY 

Use  of  electronics  and 
IT  to  further  automation 


4th  Industrial 
Revolution 


DIGITAL 

Use  of  a  digitally 
connected  end-to-end 
enterprise 


I 

1800 


I  I  I  I  I  I  I 

1900 

Traditional  Models  and 
Simulations  (M&S) 


Simulation  Based  Acquisition  (SB A) 


T 


T 


2000  TODAY 

Model-  1  DIGITAL 
Based  ENGINEERING 

Systems  (DE) 

Engineering 
(MBSE) 
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Digital  Engineering: 
MBSE  approach  for  DoD 


Current  State 

■  Our  workforce  uses  stove-piped  data  sources  and  models  in  isolation  to  support  various 
activities  throughout  the  life-cycle 

■  Current  practice  relies  on  standalone  (discipline-specific)  models 

■  Communication  is  through  static  disconnected  documents  and  subject  to  interpretation 


Future  State 


■  Digital  Engineering  moves  the  engineering  discipline  towards  an  integrated  model-based 
approach 

■  Through  the  use  of  digital  environments,  processes,  methods,  tools,  and  digital  artifacts 

■  To  support  planning,  requirements,  design,  analysis,  verification,  validation,  operation, 
and/or  sustainment  of  a  system 

■  Digital  Engineering  ecosystem  links  our  data  sources  and  models  across  the  lifecycle 

■  Provides  the  authoritative  source  of  truth 


iReQC/zre/n  Gn  ts 


Assemblies 


Subsystems 

mi 


Others 


IVIaterial 

Solution 

Analysis 


Technology 
Maturation  and 
Risk  Reduction 


Engineering  and 
Manufacturing 
Development 


Production  and 
Deployment 


Operations 
and  Support 


Manufacturing 

Systems 


Configuration  Test 

Management 


Current:  Stove-piped  models  and  data  sources  Future:  Digital  Engineering  Ecosystem 
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CREATE  Products  in  Digital 
Engineering  Context 


Digital  Engineering 

•  Digital  Engineering  vision 
moves  the  engineering 
discipline  towards  an  integrated 
model-based  approach  through 
the  use  of  digital  environments, 
processes,  methods,  tools,  and 
digital  artifacts 

•  Model  is  a  representation  of 
reality 

-  Model  is  ‘composed  of  data,  algorithms 
and/or  processes 

-  Computable  or  used  in  a  computation 


CREATE 

CREATE  program  develops  and 
deploys  validated  physics-based 
High  Performance  Computing 
(HPC)  applications  to  enable 
DoD  engineers  to  implement  and 
execute  the  digital  engineering 
paradigm  for  major  DoD 
platforms  (naval,  air,  &  ground 
vehicles  and  RF  antennas) 

Includes  ability  to  construct  and 
improve  digital  product  models 
for  weapon  platforms 

-  Tools  address  all  stages  of  the 
acquisition  process 
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Digital  Engineering  Relationships 


Physics-based  / 
Engineering 
Design  Toois 


Computational  Research  and 
Engineering  Acquisition  Tools  and 
Environments  (CREATE) 


Worid-ciass 
Computationai 
Resources  (High 
Performance 
Computing),  Software, 
Networking 

DOD 

HEC 

MODERNIZATION  PROGRAM 


Supporting  toois: 
(Large  Tradespace 
Anaiytics  datasets, 
Anaiysis  of 
Aiternatives,  Virtuai 
Prototyping 
Evaiuation,  etc.) 

ENGWRESIllEKTSYSeS 

1  "vnr 


User  selected  and  integrated  based  on  outcome  needed 


Other 

initiatives 


Traditionai 

Mod/Sim 

Soiutions 


(DoD)  Modeling  and 
Simulation  Coordination 
Office  (DMSCO) 
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Transitioning  S&T,  T&E  and  Corporate 
Knowledge  to  Engineering  &  Acquisition 


Historical  HPCMP  User  Community  Target/Expanded  HPCMP  User  Community 
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DRAFT  Vision  for  ERS,  CREATE,  et  al 
(crossing  the  Vailey  of  Death) 


DRAFT 


DRAFT 


Current  Domains:  Air  (Fixed  &  Rotary),  Surface,  Subsurface,  Ground,  RF,  Meshing,  Geometry 
Future  Domains:  Space,  Flypersonics,  Improved  Turbine  Engine,  EW,  Directed  Energy,  Others? 


/bV 

/K 

IOC 

FOC 

JCIDS  —  ICD,  CDD,  CPD  Materiel  Solution 

Technology  Maturation 

Engineering  &  Manufacturing 

Production  and 

Operations  and 

AoA  -  Guidance/Plan  Analysis 

&  Risk  Reduction 

Development 

Deployment 

Support 

Current  ERS  Uses 

Future  ERS  Uses 

1 

Future  CREATE  Uses 


EC&P  use  of  ERS,  CREATE  and  other  tools  and  environments 


Proof  of  Principle  Prototypes 


Fieldable  Prototypes 


Pre-EMD  Prototypes 


Current  =  []  B  Future  = 


DT&E  use  of  ERS,  CREATE  and  other  tools  and  environments 


Other 


Future  ERS  Use:  Industry 


I  Force  Effectiveness/Mission  models 


Force  Eff  /  Msn  Models 


Engineering  Models 


I  Eng  Models 


System  CONORS 


System  CONORS 


Digital  System  Model  /  Digital  Thread 


1  Digital  Twin 

1  CAD  /  CAM  /  Add  Mfg  | 
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Digital  Engineering  Strategy: 

Five  Goals 


Formalize  the  development,  integration 

and  use  of  models  to  inform  enterprise  and 
program  decision  making 


Provide  an  enduring  authoritative  source 
of  truth 


Incorporate  technological  innovation  to 

improve  the  engineering  practice 


Establish  supporting  infrastructure  and 
environments  to  perform  activities,  collaborate, 
and  communicate  across  stakeholders 


Transform  a  culture  and  workforce  that  adopts 
and  supports  Digital  Engineering  across  the  lifecycle 


Drives  the  engineering  practice  towards  improved  agility,  quality,  and  efficiency, 

resulting  in  improvements  in  acquisition 
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Goal  #1 :  Formalize  Development, 
Integration  &  Use  of  Models 


CREATE  in  DE  Goal  1 ; 


Specialty  Engineering  Models 


Product  i 
Support  I 
Models  ^ 


Management 

Models 


System 

Models 


Design 

Modeis 


Verification  and 
Validation  Models 


Manufacturing 

Models 


Authoritative  \ 
Source  of  i 
Truth  Ji 


Develop,  deploy  and  support 
physics-based  software  applications 
that  enable  DoD  engineers  to  rapidly: 

•  Develop  digital  product  models 
(virtual  prototypes)  for  weapon 
systems  which  can  be  used  to 
populate  design  spaces 

•  Analyze  the  performance  of  the  of 
the  systems,  using  medium-  and 
high-fidelity  physics-based  HPC 
tools,  identifying  and  fixing  system 
design  defects  and  performance 
shortfalls  thus  reducing  rework,  and 
costs,  risks,  and  schedule,  and 


Key:  Data  .1^ 


improving  performance  for  all  stages 
of  the  acquisition  process 


Models  as  the  cohesive  element  across  a  system’s  lifecycle 
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Goal  #2:  Provide  an  Authoritative 

Source  of  Truth 


Stakeholders 


•  Auditing 

•  Business  -  Cost  Estimating 

•  Business  -  Financial 
Management 

•  Contracting 

•  Engineering 

•  Facilities  Engineering 

•  Industrial  Contract  Property 
Management 

•  Information  Technology 

•  Life  Cycle  Logistics 

•  Production,  Quality,  & 
Manufacturing 

•  Program  Management 

•  Purchasing 

•  Science  &  Technology 
Management 

•  Test  and  Evaluation 


CREATE  in  DE  Goal  2: 

•  Develop  and  deploy  verified  and  validated  physics-based  HPC  tools  that  include:  all 
important  effects,  accurate  solution  algorithms,  and  model  the  complete  system  i.e. 
everything  needed  to  accurately  predict  the  performance  in  short  enough  compute 
times  for  parameter  studies 


Right  information,  right  people,  right  uses,  right  time 
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Goal  #3:  Incorporate 
Technological  Innovation 


Big  Data  and  Analytics 
<♦  Cognitive  Technologies 
Computing  Technologies 
❖  Digital-to-Physical  Fusion  Technologies 


CREATE  in  DE  Goal  3: 

•  HPCMP  eco-system  employs 
innovative  technologies  (High 
Performance  Computers,  high  speed 
networks  and  advanced  software). 

•  DoD  engineers  develop  innovative 
systems  by  rapidly  and  efficiently 
generating  many  design  options; 
identifying  the  failures  and  successes; 
and  improvements 

•  Use  of  small  teams  to  take  risks,  fail 
early  and  quickly  in  order  to  identify 
successful  product  designs 


Harness  technology,  new  approaches,  and  human-machine 
collaboration  to  enable  an  end-to-end  digital  enterprise 
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CREATE  in  DE  Goal  4: 

•  High  Performance  Computing  Ecosystem: 

•  Subject  matter  experts  from  relevant  stakeholders 

•  Validated  and  verified  data  for  use  in  engineering  and  acquisition  activities 

•  HPC  Distributed  Resource  Centers 

•  High-bandwidth  network  (DREN) 

•  Software  applications  (CREATE  codes  now  and  in  the  future) 


Foundational  support  for  Digital  Engineering  environments 


20^^  NDIA  SE  Conference 
Oct  25,  2017  1  Page-12 


Distribution  Statement  A  -  Approved  for  public  release  by  DOPSR  10/03/2017.  Case  #  18-S-0009  applies.  Distribution  is  unlimited. 


Goals  #5:  Transform 
Culture  and  Workforce 


Communication 
and  Engagement 


Leadership  Training  and 
Education 

Strategy  and  Continuous 

Implementation  Improvements 


Culture  and  Worl 
Enablers 


CREATE  in  DE  Goal  5: 

•  HPCMP  Partnerships  with  Service 
Engineering  Organizations 

•  Development  and  use  of  CREATE  builds 
computationally  skilled  DoD  workforce 

•  Training  and  support  is  provided  for  those 
accessing  CREATE  -  over  1 80  DoD 
organizations  with  -1400  users. 

•  CREATE  software  is  being  incorporated  into 
Service  Academy  and  other  university 
curricula 

•  Regular  release  of  upgraded  software 
capability 


Institutionalize  Digital  Engineering  across  the 

acquisition  enterprise 
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There  Is  Much  More  to  Do... 


•  Publish  the  Digital  Engineering  Strategy 

-  Support  development  of  implementation  guidance/direction  in  Services/Agencies 

-  Follow  with  policy? 

•  Finish  the  Digitai  Engineering  Starter  Kit 

-  Continue  development;  share/obtain  feedback  on  digital  artifact  use 

•  Engage  with  Acquisition  Programs 

-  Establish  criteria  for  use  of  Digital  Engineering  artifacts  for  decision  points 

•  Update  Competencies  across  Acquisition  Curricuia 

-  Identify  Digital  Engineering  education  and  training  outside  of  acquisition  curricula 

•  Update  Poiicy  and  Guidance  (Engineering,  et  ai) 

-  Develop/update  governance  processes,  policy,  guidance  and  contracting  language 

•  Transform  Acquisition  Practice 

-  Engage  acquisition  users 

-  Incorporate  rigor  from  Digital  Engineering  practices  and  artifacts  into  system  lifecycle 


activities 
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Systems  Engineering: 
Criticai  to  Defense  Acquisition 


Defense  Innovation  Marketplace 

http:/ /WWW.  defenseinno  vationmarketplace.  mil 


DASD,  Systems  Engineering 

http://www.acq.osd.mil/se 
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For  Additional  Information 


Ms.  Philomena  Zimmerman 
ODASD,  Systems  Engineering 

571  -372-6695 

philomena.m.zimmerman.civ(g)mail.mil 


20^^  NDIA  SE  Conference 
Oct  25,  201 7  I  Page-16 


Distribution  Statement  A  -  Approved  for  public  release  by  DOPSR  10/03/2017.  Case  #  18-S-0009  applies.  Distribution  is  unlimited. 


NDIA  System  Engineering 

Conference 
24  October  201 7 


Benjie  Spencer 

Chief  Engineer,  NOAA/National  Weather  Service 


NOAA  is  an  agency  that  enriches  iife 
through  science.  Our  reach  from  sun  to 
seafioor  heips  to  keep  citizens  informed  of 
the  changing  environment  around  them. 


Mission:  Science,  Service,  &  Stewardship. 

To  understand  and  predict  changes  in 
ciimate,  weather,  oceans,  and  coasts. 
To  share  that  knowledge  and  information  with 

others,  and 

To  conserve  and  manage  coastal  and  marine 

ecosystems  and  resources. 


National 
Environmental 
Satellite  Data 
&  Information 
Service 
(NESDIS) 


National 

Marine 

Fisheries 

Service 

(NMFS) 


Office  of 
Marine  and 
Aviation 
Operations 
(OMAO) 


National 
Weather 
Service  (NWS) 


Oceanic  and 
Atmospheric 
Research 
(OAR) 


National 

Ocean 

Service  (NOS) 
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^  NOAA’s  unique  assets  support  our 
egratedmisslon  '“~— 


NOAA  professionals 
•  20,000  staff 

12,500  FTE 
~  230  Engineers 
NOAA  Corps  -  the  Nation’s 
Uniformed  Service 


Observing  Systems 

•  ~125  weather  radars 

•  10  satellites 

•  3  buoy  networks 

•  210  tide  gages 


7,500  contractors 

18  National  Labs  &  Science 

Centers 

High  Performance  Computing 


Ships  and  Aircraft 

•  16  ships 

•  9  aircrafts 


•  5  supercomputers 


Okeanos  Explorer  NOAA  G4  and  P3 


NOAA  Employee  Operating  A  WPS  TAO  Buoy 


GOES 
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Achievements 


For  a  10  year  period  from  2005  to  2015,  the  overall 
comms  availability  was  97%  due  to  serve  weather 
Implementation  of  4G  and  VSAT  Back  Up  Restores 
availability  to  99.99%  Reducing  Downtime 

•  Commercial  T 1  and  Frame  Relay  service  with  auto  fail-over  (DoD  and  FAA  radar  data) 


Phased  implementation  approach 

•  Networx  contract  extended  in  March  201 7 


•  Comms  contract  rebid  in  2020  (unknown  impact) 

•  NEXRAD  Software  update  in  Build  18  to 

improve  link  stability  &  status  reporting 

84  sites  installed 

1 1  NWS  VSAT 
46  NWS  4G 
21  DoD  4G 
4  FAA  4G 
1  FAA  VSAT 
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Jojnt  NWS/DoD^adar  Deployment 

toPueRoTTOo^^' 

Hurricane  Maria  severely  damaged  the  FAA’s  WSR-88D  Doppler  Radar  in  PR.  NWS, 
through  the  FEMA  NRCC,  requested  DoD  support  to  deploy  two  USMC  tactical 
Doppler  radars  to  re-establish  coverage.  The  USMC  radars  were  selected  because  of 
their  ability  to  export  NEXGEN  Level  3  data. 

With  the  support  of  the  Navy  PEG  C4I  PMW  120,  Navy  SPAWARS,  Pacific, 

NORTHCOM,  MARFORNORTH,  and  USMC  2MEF,  an  unprecedented  joint 
engineering  effort  began  to  bring  the  X-band  radar  data  into  the  NWS  Advanced 
Weather  Interactive  Processing  System  (AWIPS,  the  primary  forecasting  support 
system  for  the  NWS).  The  radars  will  be  connected  to  NWS  VSAT  units  to  move  the 
data  into  the  NWS  system  where  it  can  be  utilized  by  forecasters  in  San  Juan  or  at 
back-up  forecast  offices  to  provide  life-saving  forecasts  and  warnings. 

On  21  Oct  17,  Marine  forecasters  and  technicians  will  arrive  with  the  radars  in  PR. 

They  will  link  up  with  SPAWARS  and  NWS  Radar  Operations  Center  technicians  to 
establish  the  two  sites  and  begin  the  final  efforts  to  assimilate  the  radar  data  into  the 
NWS  AWIPS.  NWS  will  also  support  interim  communications  from  the  FAA’s  Terminal 
Doppler  Weather  Radar  to  the  NWS  AWIPS  system  to  enable  forecasters  to  utilize  it 
for  forecasts  and  warnings. 
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Tail  Doppler  Radar  Upgrade 

•  Completely  dual  system  (Xmtrs, 
Rcvrs,  Processors)  for  higher 
along-track  resolution  and 
redundancy 

•  8  KW  Solid  State  Power  Amplifiers 
for  improved  sensitivity 

(5dBZ  ->  -9dBZ) 

•  Upgraded  processors  are  the 
same  as  used  in  NOAA's  NEXRAD 
WSR-88D  ground  radar 

•  Replacement  antenna  motors  to 
double  rotation  speed  and  along- 
track  resolution 


Nav.  Data 
(RINU  &  GPS) 
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N42RF  TDR  Captures  FO  Tornado  Data  on  Vortex-SE  Mission  Flight 


FO  Tornado  from  Ground  Spotter 


Reflectivity,  showing 
very  heavy  rain  and  a 
strong  inflow/updraft 
from  the  right 


Doppler  Velocity  -  Brown/orange 
away  from  aircraft  and  green/blue 
toward  plane.  Tornadic  signature 
Is  where  the  velocity  direction 
changes 


New  TDR  system  is  collecting  research  and 
operational  data  with  higher  sensitivity  and  resolution 
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turn 


Transition  to  Operations 


14=Bfl£asueem0nt  System 


Mount  Designs 


Laboratory  Test  Procedure  and  Facility 


1)  Fixed  Target  -  Resolution  Verification 

2)  Time  Response  Verification 

3)  Sensor  Offset  Derivation 

4)  Dynamic  Liquid  Tare  test 

5)  Range  Accuracy  Verification 


Saltdron*  Wat«r  T«mp«rotura  (color  bar)  and  Fur  Soal  Tracks  (magenta) 
24-MAY-20ie  to  03-SEP-2016 
with  eorller  Max.  Ice  Extent  on  05-APR-2016 


pC02i:1.a±75MMm 

(mMiHlSOf 


2017  Missions 


SAILDRONE 


iMOORING 


Saiidrone  2017:  Interdiscipiinary  Ocean 
Observations  from  the  Arctic  to  the  Troi 


rvinn 


Christian  Meinig'*,  Edward  Cokeiet'*,  Meghan  Cronin^,  Jessica  Cross'*,  Aiex  De  Robertis^,  Richard  Jenkins'*,  Carey  Kuhn^,  Noah 
Lawrence-Siavas^,  Caivin  Mordy^,  Phyiiis  Stabeno^,  Adrienne  Sutton^,  Dongxiao  Zhang^,  Jessica  Crance^,  Jennifer  Keene^,  Stacy 
Maenner'*,  Heather  Tabisoia^ 

^s/PMEL,  ^NOAA/Alaska  Fisheries  Science  Center,  '^Saiidi 


2017  Bering  Sea  &  cliulf(c1ii‘^l6ilfssfoi 

■  3  Autonomous  Surface  Vehicles  (ASVs) 

■  2  integrated  with  Autonomous  Surface  Vehicle  PCO2 
(ASVCO2)  sensor  for  Northern  Chukchi  Integrated  Study 

■  1  integrated  with  EK-80  echosounder  for  walleye  pollock 
and  northern  fur  seal  study  and  passive  acoustics 

■  ~3  month  mission 

■  Deploy  and  recover  from  dock  in  Dutch  Harbor,  AK 


2017  Tropicai  Pacific  Mission 

■  2  Autonomous  Surface  Vehicles  (ASVs)  integrated  with: 
Autonomous  Surface  Vehicle  PCO2  (ASVCO2),  ADCP,  Heat 
Flux  Sensor 

■  Participation  in  NASA  SPURS  II  Field  Campaign 

■  Climate  quality  comparison  with  instrumentation  on 
ships,  buoys  and  other  platforms 

■  ~6  month  mission 


Deploy  and  recover  from  dock  in  Alameda,  CA 


Ocean  Current  Profiling: 


Carbon:  TRL  7 


PM  EL  developed  ASVCOj  system 
measures  pco2,  pH 

1-hour  values  transmitted  via  Iridium  in 
near  real  time 

Compares  favorably  with  ship  and 
mooring  observation  testing  completed 
off  California 


Teledyne  RDI  Workhorse  300  kHz 
Dual  GPS  &  Vectornav  IMU 
Compares  favorably  with  mooring 
observation  testing  completed  off 
California 


Saiidrone  Gen  4 


2016  Mission  Resuits 

Oceanography:  TRL  5-9 


Marine  Mammal  Acoustics:  TRL  6 


Fur  Seal  Tracking:  TRL  7 


Saiidrone  observations  of  pollock  schools 


POLLOCK  AGGREGATION 


SEA  FLOOR 


Acousondes  recorded  201  of  206  mission  days  and 
obtained  ~5150  hours  of  recordings 
Saildrones  spent  69  days  within  right  whale  critical 
habitat  area  and  12.5  days  at  two  mooring  locations  for 
baseline  acoustic  comparisons 

Successful  acoustic  detection  of  killer  whale  with  possible 
detection  of  right,  fin  and  humpback  whale(s)  I 


Measured  14  atmospheric  and  oceanic 
parameters 

1-Hz  sampling  with  1-minute  data 
Transmitted  via  Iridium  in  near  real  time 
Compares  favorably  with  ship  and  mooring 
observations 


Continuously  measured  fish  acoustic  backscatter 
with  Kongsberg/Simrad  AS  echosounder 
High-quality  measurements  at  wind  speeds  less 
than  20  knots 

Comparisons  with  research  vessel  indicate  that 
shallow  pollock  react  to  ship  noise 


Tracked  30  satellite-tagged,  adult-female 
fur  seals  as  they  foraged  over  ~70  days 
Saildrones  spent  65  days  covering  fur- 
seal  grid  ~2  times 

Followed  and  recorded  behavior  and  prey 
field  of  2  fur  seals  for  1.3  and  2  days 


RESEARCH 


FISHERIES 


Acknowledgements:  This  program  is  a  muiti-institutionai  effort  and  we  thank  all  the  teams  of  contributors  in  supporting  the  design,  development,  and  operations  of  these  missions  towards  our 
common  goals.  We  thank  the  officers  and  crew  of  the  NOAA  ship  Oscar  Dyson  and  Bell  Shimada  for  their  invaluable  assistance  during  the  Saiidrone  comparisons.  This  work  is  funded  by  NOAA-OAR  and  CPO. 
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Challenges 


Replace  360  degree  scanning 
Lower  Fuselage  Weather 
Radar  with  AN/APY-11 
Multimode  Radar  System 


ela 


e  Radar  Upgrade 


16  16 


Transition  to  operations  and  any  remaining  cal/val  of  the  instruments  and 
products,  especially  the  Magnetometer 
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Air  and  Missile  Defense  Radar 

(AMDR) 


“Sea  Power  to  the  Hands  of  Our  Sailors” 

Presented  by: 

CAPT  Seiko  Okano 
Major  Program  Manager  (MPM) 

PEO  IWS  2.0  Above  Water  Sensors 


DISTRIBUTION  STATEMENT  A:  ADDroved  for  oublic  release:  Distribution  Unlimited 


AMDR  Background 


Maritime  Air  and  Missile  Defense  Joint  Forces 
(MAMDJF)  Analysis  of  Alternatives  (AoA)  results: 

□  very  large  phased  array  radar  (SPY  +30dB)  to  be  paired 
with  a  newly  constructed  combatant  to  meet  the  stressing 
BMD  and  cruise  missile  threats 


The  Next-Generation  Cruiser  Program  (CG(X)) 
was  the  planned  combatant  for  AMDR, 

2009  -  a  Radar/Hull  Study  was  conducted 

□  smaller  AMDR  could  be  paired  with  the  DDG  51  hull  and 
still  meet  these  lAMD  requirements 


USN  canceled  the  CG(X)  program,  and  restarted 
the  DDG  51  shipbuilding  program. 

New  DDG  51  configuration  with  AMDR  became 
known  as  DDG  51  Flight  III 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release;  Distribution  Uniimited. 


2016-04-13  1200 
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AMDR  Challenges 
Hardware  Systems  Engineering 


30’ Array 
(SPY+30dB) 


14’ Array 
(SPY+15dB) 


Scalability  and  Modularity 

□  IWS  2.0  partnered  with  ONR,  OSD  Title  lll/ManTech  Offices,  and  Industry  in 
an  effort  to  make  AMDR  modular,  scalable,  affordable,  and  to  reduce  risk 

Risk  reduction  Investments: 

□  Gallium  Nitride  (GaN)  Power  Electronics 

■  OSD  Title  III 

-  Conformal  Hermetic  Coating  for  Microelectronics 

-  GaN  on  SiC  MMIC  Production  for  S  and  X-band  Radar/EW 
Systems 

■  Conducted  follow-on  ManTech  GaN  Producibility  programs 

□  Digital  Array  Radar  (DAR) 

■  ONR  Future  Naval  Capability  (FNC):  Provided  an  active  phased 
array  radar  that  incudes  the  digital  beamforming  (DBF)  architecture. 

□  Affordable  Common  Radar  Architecture  (ACRA) 

■  ONR  FNC:  Provided  a  modular  and  open  combat  system  interface  to 
integrate  with  the  Product  Line  Architecture  (PLA) 

□  Affordable  Electronically  Scanned  Array  Technology  (AESAT) 

■  ONR  FNC:  Provided  electronic  components  to  reduce  lifecycle  costs 
in  the  next-generation  active  ESA  radars 

-  Components  included:  High  Power/Efficiency  MMICS  and  RF 
Power  Amplifiers,  Low  Noise  Digital  Tx/Rx  components,  and 
DBF  components 


■  Open  architecture  (OA)  standards,  interfaces,  and  equipment  were 
impiemented  into  initiai  design  for  the  radar  front-end  arrays, 
eiectronics  and  back-end  processing 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release;  Distribution  Uniimited. 


2016-04-13  1200 


3 


AMDR  Hardware  Systems  Engineering 


■  An  active,  digital  radar  enables  multiple  and 
simultaneous  high-fidelity  radar  beams  for  a 
rapid  volumetric  search 

■  Implementation  of  the  modular  hardware  and 
advancements  in  R&D  achieved  the 
following  radar  system  and  performance 
benefits: 

□  Eased  the  Systems  Engineering  Workload 

■  Decreased  the  complexity  of  the  radar 
design 

□  Improved  the  integration  and  testing  of  the  radar 
system 

□  Active  Performance 

■  Improved  detection  sensitivity 

■  Improved  clutter  attenuation 

□  SS  Reliability 

■  Improved/Increased  Mean  Time  Between 
Failure  (MTBF) 

-  10^  (100  Million)  hours 

■  Graceful  Degradation  Performance 

□  Enables  Digital  Beamforming  (DBF)  Architecture 

■  Cost  Savings  applied  to  the  acquisition 
program 

□  Sustainment  and  Lifecycle  costs  also 
decrease 


I'NC  L\SSIFIED  TOR  OmCL\L  USE  OMY 

AN/SPY-6(V)1  Overview 


Common  Array  Cooling 
System  (2  Unts) 

Cooling 
Subsystems 

DISTRBUnON  STATEMENT  D:  Dtstribubon  authorized  to 


AMDR  (AN/SPY-6)  Hardware  Overview 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release;  Distribution  Uniimited. 


2016-04-13  1200 
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AMDR  Systems  Engineering 
Finai  AMDR  Array  Design 


Scalable  AMDR 

Aperture  and  Sensitivity  to  Meet  Mission 


Intesprsted  Defe^nse  Systems 


(front  view) 


1  RMA 


DISTRIBLI  no  IMA: 


Radar  Modular  Assembly 
(RMA) 


DDG-51  AMDR 


>  37^  RMA 


Low  NRE  toscaieto  a  new  Radar 

-  Mectranicsl  Structure 

-  Common  Software 


Approved  for  Public  F^leias>e  —  Rerference  [4] 


SPY-1  -K25dB 


■  Each  RMA  measures  2’  x  2’  x  2’ 

□  Each  RMA  is  essentially  an  individual  radar 

■  This  common  architecture  ensures  the  radar’s  extensibility  and  scalability  to  other  platforms,  and  their 
particular  mission  requirements 

□  EASR  is  a  derivative  of  AMDR  that  will  be  installed  on  CVNs  and  Amphibs 

■  Common  and  Open  front/back-end  architectures  ensure: 

□  Low  NRE  for  future  radar  derivatives(radar  scaling) 

□  Common  Logistics,  Spares,  Manning,  and  Training 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release;  Distribution  Unlimited. 


2016-04-13  1200 
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DDG  51  Flight  IIA 


DDG  51  Flight  III 


■  AMDR-S  will  acquire  and  track  a  target  half  the  size  and  at  twice  the  range  compared  to  the 
AN/SPY-1,  providing  increased  flexibility  in  ship  operating  location 

■  Ability  to  react  to  and  provide  engagement  data  for  the  stressing  Very  Low  Observable/Very 
Low  Observable  Flyer  (VLO/VLOF)  target  in  a  dense  clutter  environment 

■  Capable  of  operating  in  natural  and  man-made  environments  to  meet  multi-mission 
requirements. 


AMDR  is  in  deveiopment  to  support  robust  lAMD  (BMD  and 

AAW)  Raid  Capabiiity 


DiSTRiBUTiON  STATEMENT  A:  Approved  for  pubiic  reiease;  Distribution  Uniimited. 


2016-04-13  1200 
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CTE.Lt 


AM  DR  Software  Engineering 

Radar-Combat  Integration:  Open  SW  Standards 


■  Apply  Product  Line  Architecture  (PLA) 
principles  to  create  common,  open  interfaces 
to  enable  integration 

□  Allows  future  radars  the  ability  to  integrate  with  other 
combat  systems 

□  Allows  the  USN  to  have  3^^  party  vendors  develop 
and  integrate  additional  capability  into  the  radar  and 
combat  system. 

■  Integration  of  SPY-6  into  AEGIS 

□  Relied  on  a  “modified”  B/L  9  ACS  and  the  AEGIS 
Common  Source  Library  (CSL) 

■  Developed  new  components  and  new  interfaces 

□  Demonstrated  successful  simulation  of  the  AAW  and 
BMD  Fire  Control  Loops 

□  Significant  ROI  for  B/L  1 0  (ACB-20)  for  future 
integration  and  testing 

□  Significant  reduction  of  NRE  for  integration/testing 
into  other  combat  systems  (e.g.  SSDS) 


Common  Source 
Library 


^Fix  Once.,, 

Use  Many  Times^ 

Key  Elements  of  Common 
Development: 

•  Common  Mission 
Capabilities 

•  Single  Set  of  Specifications 

•  Common  Program  Plans 

•  Single  Set  of  Processes  & 
Metrics 


•  Integrated  Team  Structure 

•  Enterprise  Products 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release;  Distribution  Unlimited. 


2016-04-13  1200 
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QUESTIONS? 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release;  Distribution  Uniimited. 


2016-04-13  1200 
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“Sea  Power  to  the  Hands  of  Our  Sailors” 


DISTRIBUTION  STATEMENT  D:  Distribution  authorized  to  DOD  and  DOD  contractors  onlv:  Other  requests  must  be  referred  to  PEO  IWS. 


Backups 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release;  Distribution  Uniimited. 


2016-04-13  1200 
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Executive  Panel: 

Interagency  Systems  Engineering 

Moderator: 

Ms.  Kristen  Baldwin 

Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering  (Acting) 

Panel  Members: 

*  Mr.  Jon  Holladay 

-  Technical  Fellow  for  Systems  Engineering 

National  Aeronautics  and  Space  Administration  (NASA) 

-  Mr.  Kent  Jones 

-  Assistant  Deputy  Administrator  for  Systems  Engineering  and  Integration 
Defense  Programs,  National  Nuclear  Security  Administration  (NNSA) 

-  Mr.  Joseph  Post 

-  Deputy  Director,  Systems  Engineering  and  Integration 
U.S.  Federal  Aviation  Administration  (FAA) 

-  Mr.  Albert  "Benjie"  Spencer 

-  Chief  Engineer  and  Director  of  Engineering  Standards  Division 
National  Oceanic  and  Atmospheric  Administration  (NOAA) 

-  Mr.  James  Tuttle 

-  Chief  Systems  Engineer,  Science  and  Technology  Directorate 
Department  of  Homeland  Security  (DHS) 


NDIA  20***  Annual  Systems  Engineering  Conference 


Panel  Discussion: 
NASA  Systems  Engineering 


Jon  B.  HoHaday 

NASA  Technical  Fellow,  Systems  Engineering 

October 24,  2017 


Key  2017  Systems  Engineering 
A  ccompiishments 

•  Engaged  Systems  Engineering  Capability  Leadership  Team  toward: 

-  Understanding  of  state  of  discipline  via  deep  dive  assessment 

-  Aligning  capability  needs  across  Centers  and  Missions 

-  Optimizing  capability  vector  focused  thru  both  tactical  and  strategic  domains 


•  Completed  Model-Based  System  Engineering  (MBSE)  Pathfinder  Part  2: 

-  Increase  stakeholder  involvement,  horizontally  and  vertically 

-  Demonstrate  applications  across  product  life-cycle 

-  Engage  the  future  state  of  NASA  Systems  Engineering 


Architecture 


Element 


Advanced 

Manufacturing 


Mission 
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Systems  Engineering 
Area  of  Emphasis  for  2018 


Expand  utilization  of  the  new  digitai  NASA  SE  Handbook 


https://www.nasa.aov/connect/ebooks/nasa-svstems-enaineerina-handbook 


Compiete  NASA  SE  Poiicy  (NPR  7123.1  B)  revision 


Commoin  T»chnkal  ProcesGAS 
■  SyAlAtn  Pifoduet  RaaliiAtlAn.  and  Teehnleal  ManagamAnt 


NASA 

SYSTEMS 

ENGINEERING 

HANDBOOK 


Workforce 

'Skil'la.  Cdmpalatidlia..  Taamwork. 
■Elhids,  Trainlngi,  Exparl«tnd« 


Continue  refinement  of  Agency’s  SE  Strategic  vector 

-  Focus  on  Technical  Leadership 

-  Recognize  the  complexity  and  dynamic  quality  of  environment 

-  Recognize  the  need  to  interface  and  partner  beyond  NASA 


NDIA  SE  Conference 
Program  Manager  Panel  Questions 


Panel  Theme: 

DoD  Executive  Panel:  Service  and  Agency  Program  Managers  discussion: 

“Teaming  with  systems  engineering  to  shape  and  control  risk,  manage  issues,  and  seize 
upon  opportunities  to  deliver  superior  warfighting  capabilities.  ” 

Moderator:  Col.  David  Mclllece,  USAF 

Panelists: 

1.  CAPT  Seiko  Okano,  PEO  Integrated  Warfare  Systems  (IWS) 

2.  COL  Mike  Milner,  Armored  Multi-Purpose  Vehicle  (AMPV)  PM 

3.  Col  Edward  Hospodar,  GPS  User  Equipment  SML 

4.  Col  Amanda  Myers,  Deputy  Director,  Global  Reach  Programs;  former  C-17  SPM 

General  Systems  Engineering 


# 

Question 

Okano 

Milner 

Hospodar 

Myer 

s 

1 

What  are  you  most  proud  of  in  your  program's  SE  activities?  What 
is  your  program  doing  that  others  might  not  be  doing  to  make 
your  systems  engineering  program  successful? 

X 

Security 

Engineering 

2 

Please  tell  a  SE  Success/horror  stories/lessons  learned. 

3 

What  do  you  value  from  your  systems  engineers  now,  what  you 
would  like  to  get,  what  would  you  prefer  less  of 
(topics/communication,  etc)? 

4 

Please  discuss  your  program’s  Risk  Management  approach,  and 
its  value.  What  is  the  role/participation  of  the  prime  contractor? 

5 

What  would  you  say  is  the  most  important  issue  or  problem  for 
systems  engineers  to  understand  about  program  management? 

6 

Has  your  view/use  of  systems  engineering  changed  over  the  years 
and  across  programs  you  have  worked?  Are  there  different 
approaches  for  different  programs  vs.  one  size  fits  all? 

7 

What  observations/lessons  to  you  have  regarding  the  systems 
engineering  roles  of  the  program  office,  prime  contractors  and 
other  stake  holders  (e.g.,  technical  authorities)?  What  are  the 
strengths/weaknesses  of  how  the  roles  are  distributed  and 
executed? 

X 

Importance  of 
early  SE  and 
trade  studies 
during 

requirements 

definition 

8 

Most  of  SE  life  cycle  emphasis  is  on  the  program  development 
during  the  early  phases.  Is  there  value  and  how  can  SE  be 
extended  to  address  the  entire  lifecycle? 

Organic  Engineering  /  Risk  Management 


# 

Question 

Okano 

Milner 

Hospodar 

M 

yer 

s 

1 

Topic:  Strengthening  organic  engineering  and  other  technical 
capabilities  in  our  own  workforce: 

Question:  In  your  programs,  where  do  you  see  the  greatest 
need  for  strengthening  engineering  capabilities?  (Quantity, 
quality,  specialized  skill  sets,  etc.) 

X 

Quality  analysis 
and  verification 
vs.  SE  process 
focus 

2 

Topic:  Understand  and  mitigate  technical  risk. 

Questions: 

How  do  you  differentiate  ^‘programmatic”  risks  such 
as  a  funding  risk,  from  “technical  risks”  such  as  not 
meeting  requirements  or  software  development  risks? 
Do  you  have  more  control  over  mitigation  activities 
for  technical  risks  than  you  do  with  programmatic 
risks? 

Do  you  have  constraints  in  identifying  technical  risks 
in  your  programs?  If  so,  please  discuss. 

X 

Stakeholder 
engagement 
presents 
opportunities  to 
“revisit” 
requirements 
which  is  a 
programmatic 
risk  but  can 
yield  insight  into 
use  cases  and 
lowers  risk  to 
OT&E 

3 

Topic:  Advantages  for  programs  with  rigorous  risk  management 
practices. 

Questions: 

What  can  we  do  to  encourage  programs  managers  to 
enact  sound  risk  management  processes? 

What  common  barriers  stand  in  the  way  of  enacting 
these  processes? 

What  experiences  can  you  share  that  will  help 
programs  to  smartly  accept/manage  increased  risk  in 
order  to  achieve  greater  and/or  faster  successful 
outcomes? 

X 

Importance  of 
prototyping  and 
early  integration 
into  the 

architecture  and 
next  level  of 
assembly  for 
feedback  or  to 
determine 
missing 
requirements 

4 

Topic:  The  Department  recently  issued  an  updated  guide  for  Risk 
Management:  The  DoD  Risk,  Issue,  and  Opportunity  (aka  “RIO”) 
Management  Guide  for  Defense  Acquisition  Programs. 

Questions: 

How  have  you  applied  RIO  concepts  (such  as  Issue 
Management,  Opportunity  Management,  Cross- 
Program  Risk  Management,  etc.)  in  your  overall  risk 
mitigation  approach? 

Do  you  have  any  best-practices  or  other  experiences 
that  can  inform/improve  the  RIO  approach? 

Panelist-Suggested  Questions 


# 

Question 

Okano 

Milner 

Hospodar 

Myers 

1 

Please  feel  free  to  offer  any  questions  you  would  like  included. . . 

Question:  «text  here» 

Executive  Panel: 

DoD  Systems  Engineering 

Panel  Members: 

*  Ms.  Kristen  Baldwin 

-  Acting  Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering 

*  Mr.  Leo  Smith,  USA 

*  Division  Chief,  Systems  Engineering  Program  Support 
ASA(ALT)  System  of  Systems  Engineering  and  Integration 

*  Mr.  William  Bray,  USN 

-  Deputy  Assistant  Secretary  of  the  Navy  for  Research,  Development,  Test 
and  Evaluation 

*  Col  Laird  Abbott,  USAF 

-  Chief,  Engineering  and  Force  Management  Division,  Deputy  Assistant 
Secretary  of  the  Air  Force  for  Science,  Technology  and  Engineering 
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DoD  Systems  Engineering 

Opportunities 

Ms.  Kristen  Baldwin 

Acting  Deputy  Assistant  Secretary  of  Defense 
for  Systems  Engineering  (DASD(SE)) 

20th  Annuai  NDiA  Systems  Engineering  Conference 
Springfieid,  VA  |  October  24,  2017 
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Defense  Research  &  Engineering 

Strategy 


Mitigate  current  and  anticipated  threat  capabilities 


Enable  new  or  extended  capabilities  affordably 
in  existing  military  systems _ 

Create  technology  surprise  through  science 
and  engineering 


Focus  on  Technical  Excellence 
Deliver  Technologically  Superior  Capabilities 
Grow  and  Sustain  our  S&T  and  Engineering  Capability 
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DoD  Engineering  Focus  Areas 


*  Grow  and  maintain  engineering  and  technicai  ieadership  taient 

*  Mature  engineering  practices  to  impiement  moduiarity,  agiiity, 
and  innovation  into  systems 

*  Leverage  advanced  analyticai  and  computing  technoiogies  and 
migrate  to  digital  acquisition,  engineering  and  manufacturing 
practice 

*  Address  complex  software  development,  integration,  and 
sustainment  challenges 

*  Establish  practices  for  cyber-resilient  aerospace  and  defense 
systems 

*  Enable  trust  and  access  to  assured  hardware  and  software 

*  Implement  enterprise  and  mission  integration  management 
capabilities 
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Headquarters  U.S.  Air  Force 


Integrity  -  Service  -  Excellence 


US  Air  Force  Engineering  Enterprise 

An  Update  to  the  NDIA  SE  Conference 


Col  Laird  Abbott 
SAF/AQR 


Air  Force  Engineering  Enterprise 

Cross  Cutting  Strategic  Direction 


Key  Leadership  Focus  Areas 

Own  Technical 
Baseline 


Modeling  & 
Simuiation 


Onming  the  Technical  Baseline 

-a  Key  Enabler  ^  . 


1 

Iiliiiilliiilliliiiiliiiiii 
- 1 - - 


OpniiiMSrKMn 


Ml  MMU  IM 


Cybersecurity  .. 
^Skills  Crisis^  • 


jfli 

IhS  - 

rforceTimes  1 

System  cybersecurity  must  begin  with  engineering,  Air 
Force  general  says 


Regain  Gov’t  Control  of  Pgms 
Informed  Decision  Making 
Skill-gap  Identify/Mitigate 


Industry  Consensus  Tech  Solutions 
Open  Key-Interfaces 
Service  Oriented  Architectures 
Common  Messaging  Language 


Joint  Simulation  Environment 
Inventory  M&S  +  Develop  Regmts 
Enable  Experimentation/Prototyping 


Cyber  Campaign  Plan 
Cyber  Workforce 
Risk  Id  &  Management 
Process  &  Policy  Dev 


Breaking  Barriers  ...  Since  1947 


The  AF  EE  Challenge  Problem 

Digital  Engineering 


Given  complexity,  uncertainty  and  the 
lack  of  agility  have  diminished  Air  Force 
Acquisition’s  ability  to  meet  mission 
needs 

How  do  we  establish  an  all  digital 
authoritative  source  of  life  cycle 
technical  data  for  every  weapon  system 

In  order  to  deliver  more  capability  more 
rapidly  then  ever  before 


Digital  Engineering  » 

Owning  the  Technical  Baseline  DIGITAIIY 


OWNING  THE  TECHNICAL 


BASELINE 


>  Technical  Competence 
•  Engineering  Data 

>  Engineering  Analytic  Capability 


Breaking  Barriers  ...  Since  1947 


3 


The  End 


Interagency  Panel  -  Questions 


Executive  Panel:  Interagency  Systems  Engineering 

20th  NDIA  SE  Conference,  Waterford,  Springfield,  VA 
Tuesday,  October  24, 2017 
11:15  - 12:30  am 


Questions 


1.  It’s  been  roughly  10  years  since  INCOSE  kicked  off  their  MBSE  initiative.  The 
lAWG  just  released  a  white  paper  this  year  that  talked  challenges  of  MBSE 
infusion.  What  are  your  thoughts  on  where  are  in  the  adoption  process  and  how 
long  will  it  take  until  it’s  an  everyday  thing. 

2.  If  you  could  only  pick  one  thing  as  the  focus  to  improve  the  efficiency  of  your 
System  Engineering  response,  what  would  it  be  and  why? 

3.  What’s  the  hardest  thing  about  your  job. . . 


Program  Managers  Panel 

Moderator:  Col  David  Mclilece,  USAF 

Deputy  for  Systems  Engineering  Plans  and  Policy,  ODASD(SE) 

Panel  Members: 

*  COL  Michael  Milner,  USA 

-  Project  Manager,  Armored  Multi-Purpose  Vehicle  (AMPV) 

*  Col  Amanda  Myers,  USAF 

-  Deputy  Director,  Global  Reach  Programs 

-  Former  C-17  System  Program  Manager 

*  CAPT  Seiko  Okano,  USN 

-  Major  Program  Manager,  PEO  Integrated  Warfare  Systems 
(IWS)  2.0 

-  Col  Edward  Hospodar,  USAF 

-  Senior  Materiel  Leader,  GPS  User  Equipment  Division,  Space 
and  Missile  Systems  Center 


Col  Amanda  Myers 

Deputy  Director,  Global  Reach  Programs 
SAF/AQQ 


TfV 


Safe 

Suitable 

Effective 


Sustaining  Engineering 

•  C-17  Strategic  Goal:  Increase  focus  on  OSS&E  & 
"Own  the  Technical  Baseline" 

•  Grow  Organic  Sustaining  Engineering  skills 

•  Relook  at  engineering  processes;  ensure  proper 
alignment  between  USG  and  OEM 

•  Robust  integrity  programs 

•  Proactively  identify  watch  areas 

•  Increasing  USG  oversight/rigor 

•  Transition  from  production  driven  focus  to: 
aircraft  aging,  corrosion,  DM  S/Obsolescence 


Military  GPS  User 
Equipment 
Modernization 

NDIA 

20^^  Annual  Systems  Engineering 
Conference 

Col  Ed  Hospodar 


Chief,  GPS  User  Equipment  Division 


Global  Positioning  Systems  Directorate 
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1 966  Aerospace  Corporation 
“Navigation  Satellite  Study” 
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1 966  Aerospace  Corporation 
“Navigation  Satellite  Study” 
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US£A  Acnt/S 

[  U3£R  AA$3Hi£ 

use/t  Acr/i/£ 

USER  ACTIVE 

1-way  ranges,  passive 
receivers,  crystal 
oscillators 

Passive  (one-way) 
reduces  UE  power  and 
avoids  detection 

Internal  computer  spreads 
the  burden  for  1 ,000’s  of 
users  and  avoids  sending 
measurements 

Crystal  oscillator 
minimizes  UE  SWAP-C 
and  doesn’t  hurt  accuracy 

Autonomous  receivers 


SWAP-C  =  Size,  Weight,  and  Power  -  Cost 


The  widespread  use  of  GPS  and  duplication  by  all  other  GNSS  validate  these  choices 
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Civil  Cooperation 

•  3+  Billion  civil  &  commercial 
users  worldwide 

•  Search  and  Rescue 

•  Civil  Signals 

-  L1  C/A  (Original  Signal) 

-  L2C  (2"d  Civil  Signal) 

-  L5  (Aviation  Safety  of  Life) 

-  L1C  (International) 


Spectrum  35  Satellites  /  31  Set  Healthy 

•  World  Radio  Conference  Baseline  Consteiiation:  24  Sateiiites 

•  International  _ 


Telecommunication  Union 

Satellite  Block  Quantity  Average  Age 

Oldest 

•  Biiaterai  Agreements 

GPS  MR 

12 

15.7 

20.1 

•  Adjacent  Band  Interference 

GPS  IIR-M 

7 

10.1 

11.9 

GPS  IIP 

12 

3.6 

7.3 

Constellation 

31 

9.7 

20.1 

GPS  Overview 


Department  of  Defense 

•  Services  (Army,  Navy,  AF,  USMC) 

•  Agencies  (NGA  &  DISA) 

•  US  Naval  Observatory 

•  PNTEXCOM 

•  GPS  Partnership  Council 

Maintenance/Securitv 

•  All  Level  I  and  Level  II 

-  Worldwide  Infrastructure 

-  NATO  Repair  Facility 

•  Develop  &  Publish  ICDs  Annually 

-  Public  ICWG:  Worldwide  Involvement 

-  Materials  Available  at:  gps.gov/technical/icwg 

•  Update  GPS.gov  Webpage 

•  Load  Operational  Software  on 
over  970,000  SAASM  Receivers 

•  Distribute  PRNs  for  the  Worid 

-  1 20  for  US  and  90  for  GNSS 


Department  of  Transportation 

•  Federal  Aviation  Administration 

Department  of  Homeland  Security 

•  U.S.  Coast  Guard 


ASOFI  SEP  17 

International  Cooperation 

•  57  Authorized  Allied  Users 

-  25+  Years  of  Cooperation 

•  GNSS 

-  Europe  -  Galileo 
-China  -  Beidou 

-  Russia  -  GLONASS 
-Japan  -  QZSS 

-  India  -  NAVIC 

UNCLASSIFIED//APPROVED  FOR  PUBLIC  RELEASE 


control  sequent 


Legacy  (GPS  IIA/IIR) 

•  Basic  GPS 

•  NUDET  (Nuclear  Detonation) 
3n  System  (NDS) 


GPSIIR-M 

*  2nd  Civil  signal  (L2C) 

*  New  Military  signal 

*  Increased  Anti^am  power 


GPS  IIP 

•  3rd  Civil  Signal  (L5) 

•  Longer  Life 

•  Better  Clocks 


GPSIII(SV01-10) 


•  Accuracy  & ^wer  ft- 

•  Increased  AntiSam  power 

•  Inherent  Signal  Integrity 

•  Common  L1C  Signal 

•  Longer  Life 


GPSIII(SV11+) 

•  Unified  S-Band  Telemetry, 
Tracking  &  Commanding 
Search  &  Rescue  (SAR) 
Payload 

Laser  Retroreflector  Array 
Redesigned  NDS  Payload 
Regional  Military  Protect  (RMP) 


Ground  System 


Legacy  (OCS) 

•  Mainframe  System 

•  Command  &  Control 

•  Signal  Monitoring 


AEP 

•  Distributed  Architecture 

•  Increased  Signal 
Monitoring  Coverage 

•  Security 

•  Accuracy 

•  Launch  And 
Disposal  Operations 


OCX  Block  0 
GPS  lillaunch  &  Checkout 

GPS  III  Contingency  Ops  (COps) 

•  GPS  III  Missbn  on  AEP 


OCX  Block  1 

•  Fly  Constellation  &  GPS  III 

•  Begin  New  Signal  Control 

•  Upgraded  Information  Assurance 

OCX  Block  2+ 

•  Control  all  signals 

•  Capability  On-Ramps 

•  GPS  III  Evolution 


Modernized 

•  M-Code  Receivers 

•  Common  GPS  Modules 

•  Increased  Access/  Power  with  M-Code 

•  Increased  Accuracy 

•  Increased  Availability 
Increased  Anti-Tamper/ Anti-  Spoof 
Increased  Acquisition  in  Jamming 
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GPS  Enterprise  Operational  View 


GPS 


SPACE  AND  MISSILE  SYSTEMS  CENTER 


GPS  IIR/lIRM 


SPACE  SEGMENT 


SV  Families  Provide 

S~Band  /  L-Band  Telemetry  1 1  o  c  o 

To  Control  and  User  Segments  U  O  C  K 

SEGMENT 

Applies  Space  and  Control 
Segment  Data  for  Position, 
»  Navigation  &  Timing 


S-band  ^ 
Uplink/Downlink 


L-band  Downlink 
•^(^avigation  Services) 


Operational  Control  System  (OCS) 

Next  Generation  Operational  Control  System  (OCX)v^ 

^  #^->#rgjp#jM«Ground. , 


Alternato  Master  Control 
Station  (located  at  VAFB) 
Backup  Facility  to  MCS 


Uplinks 
data  to  SVs 


Monitor 

Stations 


Alternate 

Mes" 


Receivers  that 
continuously  collect 
GPS  data  from  all  the 
sotclHtcs  In  view 


Master  Control  Station  (operated  by 
2SOPS  atSAFB):  Responsible  for 
Tr&C,;^nj^rb|t’hoolthy  optjratiorrs^ 
(NavandjNps"  payloads)  ^  'mM 


Includes  NCA, 
AFSCN,  HANU,  ICADS, 
GIN,  USNO,  AFTAC, 
OSS,  IMOSC,  etc. 


External 

Interfaces 


CONTROL  SEGMENT 

Telemetry,  Tracking  and  Commanding  (TT&C)  of  Space  Segment  Assets 
Distributes  Data  of  Interest  to  the  External  Interfaces  of  the  User  Segment 
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MilitaryGPS  User  Equipment  (MGUE) 


•  Commercial  market-driven  acquisition  approach 


•  Three  vendors  developing  modernized  receiver  cards 


-  Ground  form  factor 

-  Aviation/Maritime  form  factor 

•  Current  Status 

-  L-3  Technologies  first  to  receive 
security  certification  Oct  2016 

-  Developmental  testing  ongoing 

-  Conducting  early  integration 
activities  to  support 
Service-nominated  Lead  Platforms 


MGUE  Components  GPS  Interface  t 
A-KIt  £20*12!  - 

Authority 


GRAM  SEM-E-M 
(GRAM  S/M) 


Lead  Platforms 

B-Kit 


Lead  Platforms 


Ground 


IS-GPS-1  53 
&  MSID 


Aviation  & 
Maritime 


t 


ARMY 

DAGR  Distributed  Device 


C4ISR  System 


MARINES 

Joint  Light  Tactical 
Vehicle  (JLTV) 


IS-GPS-1 77 


AIR  FORCE 

B-2  Spirit 


f  *■ 

1 

1 

1 

f  PNT 

1 

■qpnts  navy 

1 

Arleigh  Burke  Class 

1 

-  Destroyer  (DDG) 

Q 
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Military  GPS  User  Equipment 
Prototype  GPS  Receiver  Flight  Tested  on  B-2 

SPACE  AND  MISSILE  SYSTEMS  CENTER 


Prototype 
Miniaturized 
Airborne  GPS 


4  Successful 
B-2  Test  Flights 


Prototype 
Military  GPS 
User 

Equipment 
Receiver  Card 
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MGUE  Precision  Guided  Munitions  Test 
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•  GPS  is  the  Global  Utility 

-  Committed  to  maintaining  uninterrupted  service 

-  “The  Gold  Standard” 


•  Modernizing  to  enhance  GPS  resiliency  by: 

-  Upgrading  all  three  segments 

-  Moving  to  M-Code 

-  Adding  civil  signals 

•  Exploring  multi-GNSS  potential 


Perspectives 


Deliver  capabilities,  execute  with  excellence,  lead  with  transparency 
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